Robotically controllable field generators for registering multiple coordinate frames

ABSTRACT

Certain aspects relate to systems with robotically controllable field generators and applications thereof. For example, a robotic medical system may include a first robotic arm that is configured to couple to an electromagnetic (EM) field generator. The first robotic arm be capable of moving the EM field generator. The robotic medical system may also include one or more processors. The processors may determine an EM position of an EM sensor within the EM field in an EM coordinate frame associated with the EM field generator. The processors also determine a position of the EM field generator in a robotic coordinate frame associated with the first robotic arm. The processors determine a registration between the EM coordinate frame and the robotic coordinate frame based on the position of the EM field generator. Based on the registration, the processors may determine a position of the EM sensor in the robotic coordinate frame.

RELATED APPLICATIONS

This application claims priority to following, the disclosures of whichare hereby incorporated by reference in their entirety: U.S. ProvisionalApplication No. 63/070,472, filed Aug. 26, 2020, and entitledROBOTICALLY CONTROLLABLE FIELD GENERATORS FOR ROBOTIC MEDICAL SYSTEMS;U.S. Provisional Application No. 63/084,979, filed Sep. 29, 2020, andentitled ROBOTICALLY CONTROLLABLE FIELD GENERATORS FOR ROBOTIC MEDICALSYSTEMS; U.S. Provisional Application No. 63/084,950, filed Sep. 29,2020, and entitled ROBOTICALLY CONTROLLABLE FIELD GENERATORS FOR ROBOTICMEDICAL SYSTEMS; and U.S. Provisional Application No. 63/216,990, filedJun. 30, 2021, and entitled ROBOTICALLY CONTROLLABLE FIELD GENERATORSFOR ROBOTIC MEDICAL SYSTEMS.

TECHNICAL FIELD

The systems and methods disclosed herein are directed to fieldgenerators for robotic medical systems, and more particularly, torobotically positionable and/or controllable field generators forregistering locations across multiple coordinate frames in a medicalsystems as well as related devices, systems, and methods.

BACKGROUND

Medical procedures such as laparoscopy or endoscopy may involveaccessing and visualizing an internal region of a patient. In alaparoscopic procedure, a medical instrument can be inserted into theinternal region through a laparoscopic access port. In an endoscopicprocedure, a thin, flexible tubular medical instrument may be insertedinto the internal region through a natural patient orifice. The medicalinstrument can include an end effector configured to perform a functionduring the procedure.

In certain procedures, a robotically-enabled medical system may be usedto control the insertion and/or manipulation of the medical instrumentand end effector. The robotically-enabled medical system may include arobotic arm, or other instrument positioning device, having amanipulator assembly used to control the positioning of the instrumentduring the procedure.

The robotically-enabled medical system may be configured to determinethe position of the medical instrument based on an output of one or morepositions sensors that can be positioned on the medical instrument.

SUMMARY

A robotic medical system can include an electromagnetic (EM) fieldgenerator that is configured to couple to (or otherwise be integratedinto) a robotic arm of the system. Such EM field generators can beconsidered robotically controllable or positionable as they can becontrolled or repositioned using the robotic arm. The EM field generatorcan produce a magnetic field within which the positions of one or moreEM sensors can be determined. Because the EM field generator is coupledto the robotic arm, the kinematics of the robotic arm can be used toestablish a registration between an EM coordinate frame of the EM fieldgenerator and a robotic coordinate frame or a global coordinate frame ofthe system.

This arrangement can reduce or eliminate the need for more complexregistration steps that can require user input to establish arelationship between the EM coordinate frame and the robotic or globalcoordinate frame. This arrangement can also improve the accuracy withwhich the positions of the EM sensors are determined by, for example,improving the setup of the field generator corresponding robotic arms,tracking the medical instruments, detecting distortion that may impactthe sensors of the systems, and facilitate sensor fusion with additionalmodalities.

These and other features and advantages of the robotically controllableor positionable EM field generators will be described in more detailbelow. The systems, methods and devices of this disclosure each haveseveral innovative aspects, no single one of which is solely responsiblefor the desirable attributes disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed aspects will hereinafter be described in conjunction withthe appended drawings, provided to illustrate and not to limit thedisclosed aspects, wherein like designations denote like elements.

FIG. 1 illustrates an embodiment of a cart-based robotic system arrangedfor diagnostic and/or therapeutic bronchoscopy, according to an exampleembodiment.

FIG. 2 depicts further aspects of the robotic system of FIG. 1,according to an example embodiment.

FIG. 3 illustrates an embodiment of the robotic system of FIG. 1arranged for ureteroscopy, according to an example embodiment.

FIG. 4 illustrates an embodiment of the robotic system of FIG. 1arranged for a vascular procedure, according to an example embodiment.

FIG. 5 illustrates an embodiment of a table-based robotic systemarranged for a bronchoscopic procedure, according to an exampleembodiment.

FIG. 6 provides an alternative view of the robotic system of FIG. 5,according to an example embodiment.

FIG. 7 illustrates an example system configured to stow robotic arm(s),according to an example embodiment, according to an example embodiment.

FIG. 8 illustrates an embodiment of a table-based robotic systemconfigured for a ureteroscopic procedure, according to an exampleembodiment.

FIG. 9 illustrates an embodiment of a table-based robotic systemconfigured for a laparoscopic procedure, according to an exampleembodiment.

FIG. 10 illustrates an embodiment of the table-based robotic system ofFIGS. 5-9 with pitch or tilt adjustment, according to an exampleembodiment.

FIG. 11 provides a detailed illustration of the interface between thetable and the column of the table-based robotic system of FIGS. 5-10,according to an example embodiment.

FIG. 12 illustrates an alternative embodiment of a table-based roboticsystem, according to an example embodiment.

FIG. 13 illustrates an end view of the table-based robotic system ofFIG. 12, according to an example embodiment.

FIG. 14 illustrates an end view of a table-based robotic system withrobotic arms attached thereto, according to an example embodiment.

FIG. 15 illustrates an exemplary instrument driver, according to anexample embodiment.

FIG. 16 illustrates an exemplary medical instrument with a pairedinstrument driver, according to an example embodiment.

FIG. 17 illustrates an alternative design for an instrument driver andinstrument where the axes of the drive units are parallel to the axis ofthe elongated shaft of the instrument, according to an exampleembodiment.

FIG. 18 illustrates an instrument having an instrument-based insertionarchitecture, according to an example embodiment.

FIG. 19 illustrates an exemplary controller, according to an exampleembodiment.

FIG. 20 depicts a block diagram illustrating a localization system thatestimates a location of one or more elements of the robotic systems ofFIGS. 1-10, such as the location of the instrument of FIGS. 16-18, inaccordance to an example embodiment.

FIG. 21 illustrates an example EM field generator producing a magneticfield having a working volume. An example EM sensor, positioned withinthe working volume of the EM field generator is also shown, inaccordance to an example embodiment.

FIG. 22A is a perspective view illustrating an embodiment of a roboticmedical system that includes the EM field generator of FIG. 21,according to an example embodiment.

FIGS. 22B and 22C illustrate example registration steps that can be usedto register a robotic coordinate frame associated with the roboticmedical system of FIG. 22A with a EM coordinate system associated withthe EM field generator, according to an example embodiment.

FIG. 23 is a perspective view illustrating an embodiment of a roboticmedical system that includes an embodiment of a robotically controllablefield generator coupled to a robotic arm of the system, according to anexample embodiment.

FIG. 24 is a perspective view of an embodiment of a roboticallycontrollable field generator that is configured to couple to a roboticarm, according to an example embodiment.

FIG. 25 illustrates that the robotically controllable field generator ofFIG. 24 can be configured to couple to an instrument drive mechanism ofa robotic arm, according to an example embodiment.

FIG. 26 is a flow chart illustrating an embodiment of a method forperforming a robotic medical procedure using a robotically controllablefield generator that is configured to couple to a robotic arm, accordingto an example embodiment.

FIGS. 27A and 27B illustrate an embodiment of a robotic medical systemincluding a robotically controllable field generator configured forautomatic instrument tracking and mapping of anatomical features,according to an example embodiment.

FIGS. 28A and 28B illustrate an embodiment of a robotic medical systemincluding a robotically controllable field generator configured forfacilitating setup and placement of the field generator and expandingthe working volume thereof, according to an example embodiment

FIG. 29 is a block diagram representative of an embodiment of a roboticmedical system that includes a robotically controllable field generatorcoupled to a robotic arm of the system, according to an exampleembodiment.

FIGS. 30A and 30B are perspective views that illustrate an example ofmoving an EM field generator with a robotic arm such that an EM sensoris positioned at a predetermined position within an EM field, accordingto an example embodiment.

FIGS. 31A and 31B are perspective views that illustrate an example ofmoving an EM field generator with a robotic arm such that an EM sensoris positioned at within a predetermined region within an EM field.,according to an example embodiment

FIGS. 32A and 32B are perspective views that illustrate an example ofmoving an EM field generator with a robotic arm along a path that tracksa path of movement of an EM sensor, according to an example embodiment.

FIGS. 33A-33D are perspective views that illustrate an example of movingan EM field generator with a robotic arm such that an EM sensor movingalong a path remains positioned within a predetermined region of an EMfield, according to an example embodiment.

FIG. 34 is a flowchart providing an example method for moving an EMfield generator coupled to a robotic arm based on a determined positionof an EM sensor within an EM field, according to an example embodiment.

FIGS. 35A and 35B are perspective views that illustrate an example ofmoving an EM field generator to a field generator position using arobotic arm based on determined positions of a plurality of EM sensorswithin an EM field, according to an example embodiment.

FIGS. 36A-36C are perspective views that illustrate an example ofreadjusting a field generator position of an EM field generator using arobotic arm based on the determined positions of a plurality of EMsensors within an EM field, wherein at least one of the plurality of EMsensors is moving, according to an example embodiment.

FIG. 37 is a flowchart illustrating an example method for determining afield generator position for an EM field generator mounted on a roboticarm based on determined positions of a plurality of EM sensors within anEM field, according to an example embodiment.

FIGS. 38A, 38B, and 38C illustrate an embodiment of a robotic medicalsystem including a robotically controllable field generator that can bemoved with a robotic arm to expand a working volume of the fieldgenerator, according to an example embodiment.

FIGS. 39A-39D illustrate an embodiment of a robotic medical systemincluding a robotically controllable field generator that can be movedwith a robotic arm to detect positions of EM sensors and expand aworking volume of the field generator, according to an exampleembodiment.

FIGS. 40A-40D illustrate an embodiment of a robotic medical systemincluding a robotically controllable field generator that can be movedwith a robotic arm to track positions of EM sensors within an expandedworking volume of the field generator, according to an exampleembodiment.

FIG. 41A is a flowchart illustrating an example method for expanding aworking volume of a robotically controllable field generator, accordingto an example embodiment.

FIG. 41B is a flowchart illustrating another example method forexpanding a working volume of a robotically controllable fieldgenerator, according to an example embodiment.

FIGS. 42A-42C illustrate an embodiment of a robotic medical system witha robotically controllable field generator configured to facilitatealignment of a percutaneously insertable instrument with an EM target,according to an example embodiment.

FIGS. 43A and 43B illustrate an embodiment of a robotic medical systemwith a robotically controllable field generator configured to facilitatealignment of an instrument guide mounted on the field generator with anEM target, according to an example embodiment.

FIGS. 44A and 44B illustrate an embodiment of a robotic medical systemwith a robotically controllable field generator configured to facilitatealignment of an instrument guide with an EM target, according to anexample embodiment.

FIG. 45A is a flowchart illustrating a method for aligning apercutaneously insertable instrument with an EM target using arobotically controllable field generator, according to an exampleembodiment.

FIG. 45B is a flowchart illustrating a method for aligning an instrumentguide for a percutaneously insertable instrument with an EM target usinga robotically controllable field generator, according to an exampleembodiment.

FIG. 46A illustrates an embodiment of a robotic medical system moving anEM field generator relative to a fixed EM position sensor in order todetect EM distortion, according to an example embodiment.

FIG. 46B illustrates a comparison of a robotic trajectory and an EMsensor trajectory during the movement of the EM field generator of FIG.46A, according to an example embodiment.

FIG. 46C illustrates an embodiment of a robotic medical system moving anEM field generator relative to a fixed EM position sensor in order todetect EM distortion that can account for changes in position andorientation, according to an example embodiment.

FIG. 47 is a flowchart illustrating an example method for EM distortiondetection, according to an example embodiment.

FIG. 48 is a flowchart illustrating another example method for EMdistortion detection, according to an example embodiment.

FIG. 49A illustrates a robotic medical system with an ultrasonic probeattached thereto during a procedure for calibrating an imaging place ofthe ultrasonic probe, according to an example embodiment.

FIG. 49B illustrates a robotic medical system including an EM fieldgenerator and an ultrasonic probe. The imaging plane of the ultrasonicprobe is calibrated with respect to a robotic coordinate frame and an EMcoordinate frame such that robotic tools and EM sensor positions can beoverlaid on the imaging plane, according to an example embodiment.

FIG. 50 is a flowchart illustrating an example method for calibrating animaging plane of an ultrasonic probe for use with a robotic medicalsystem, according to an example embodiment.

FIGS. 51A and 51B illustrate example heat and point maps generated usinga depth sensor attached to a robotic arm of a robotic medical system,according to an example embodiment.

FIG. 51C illustrates an embodiment of an example depth sensor, accordingto an example embodiment.

DETAILED DESCRIPTION 1. Overview.

Aspects of the present disclosure may be integrated into arobotically-enabled medical system capable of performing a variety ofmedical procedures, including both minimally invasive, such aslaparoscopy, and non-invasive, such as endoscopy, procedures. Amongendoscopic procedures, the system may be capable of performingbronchoscopy, ureteroscopy, gastroscopy, etc.

In addition to performing the breadth of procedures, the system mayprovide additional benefits, such as enhanced imaging and guidance toassist the physician. Additionally, the system may provide the physicianwith the ability to perform the procedure from an ergonomic positionwithout the need for awkward arm motions and positions. Still further,the system may provide the physician with the ability to perform theprocedure with improved ease of use such that one or more of theinstruments of the system can be controlled by a single user.

Various embodiments will be described below in conjunction with thedrawings for purposes of illustration. It should be appreciated thatmany other implementations of the disclosed concepts are possible, andvarious advantages can be achieved with the disclosed implementations.Headings are included herein for reference and to aid in locatingvarious sections. These headings are not intended to limit the scope ofthe concepts described with respect thereto. Such concepts may haveapplicability throughout the entire specification.

A. Robotic System—Cart.

The robotically-enabled medical system may be configured in a variety ofways depending on the particular procedure. FIG. 1 illustrates anembodiment of a cart-based robotically-enabled system 10 arranged for adiagnostic and/or therapeutic bronchoscopy. During a bronchoscopy, thesystem 10 may comprise a cart 11 having one or more robotic arms 12 todeliver a medical instrument, such as a steerable endoscope 13, whichmay be a procedure-specific bronchoscope for bronchoscopy, to a naturalorifice access point (i.e., the mouth of the patient positioned on atable in the present example) to deliver diagnostic and/or therapeutictools. As shown, the cart 11 may be positioned proximate to thepatient's upper torso in order to provide access to the access point.Similarly, the robotic arms 12 may be actuated to position thebronchoscope relative to the access point. The arrangement in FIG. 1 mayalso be utilized when performing a gastro-intestinal (GI) procedure witha gastroscope, a specialized endoscope for GI procedures. FIG. 2 depictsan example embodiment of the cart in greater detail.

With continued reference to FIG. 1, once the cart 11 is properlypositioned, the robotic arms 12 may insert the steerable endoscope 13into the patient robotically, manually, or a combination thereof. Asshown, the steerable endoscope 13 may comprise at least two telescopingparts, such as an inner leader portion and an outer sheath portion, eachportion coupled to a separate instrument driver from the set ofinstrument drivers 28, each instrument driver coupled to the distal endof an individual robotic arm. This linear arrangement of the instrumentdrivers 28, which facilitates coaxially aligning the leader portion withthe sheath portion, creates a “virtual rail” 29 that may be repositionedin space by manipulating the one or more robotic arms 12 into differentangles and/or positions. The virtual rails described herein are depictedin the Figures using dashed lines, and accordingly the dashed lines donot depict any physical structure of the system. Translation of theinstrument drivers 28 along the virtual rail 29 telescopes the innerleader portion relative to the outer sheath portion or advances orretracts the endoscope 13 from the patient. The angle of the virtualrail 29 may be adjusted, translated, and pivoted based on clinicalapplication or physician preference. For example, in bronchoscopy, theangle and position of the virtual rail 29 as shown represents acompromise between providing physician access to the endoscope 13 whileminimizing friction that results from bending the endoscope 13 into thepatient's mouth.

The endoscope 13 may be directed down the patient's trachea and lungsafter insertion using precise commands from the robotic system untilreaching the target destination or operative site. In order to enhancenavigation through the patient's lung network and/or reach the desiredtarget, the endoscope 13 may be manipulated to telescopically extend theinner leader portion from the outer sheath portion to obtain enhancedarticulation and greater bend radius. The use of separate instrumentdrivers 28 also allows the leader portion and sheath portion to bedriven independently of each other.

For example, the endoscope 13 may be directed to deliver a biopsy needleto a target, such as, for example, a lesion or nodule within the lungsof a patient. The needle may be deployed down a working channel thatruns the length of the endoscope to obtain a tissue sample to beanalyzed by a pathologist. Depending on the pathology results,additional tools may be deployed down the working channel of theendoscope for additional biopsies. After identifying a nodule to bemalignant, the endoscope 13 may endoscopically deliver tools to resectthe potentially cancerous tissue. In some instances, diagnostic andtherapeutic treatments can be delivered in separate procedures. In thosecircumstances, the endoscope 13 may also be used to deliver a fiducialto “mark” the location of the target nodule as well. In other instances,diagnostic and therapeutic treatments may be delivered during the sameprocedure.

The system 10 may also include a movable tower 30, which may beconnected via support cables to the cart 11 to provide support forcontrols, electronics, fluidics, optics, sensors, and/or power to thecart 11. Placing such functionality in the tower 30 allows for a smallerform factor cart 11 that may be more easily adjusted and/orre-positioned by an operating physician and his/her staff. Additionally,the division of functionality between the cart/table and the supporttower 30 reduces operating room clutter and facilitates improvingclinical workflow. While the cart 11 may be positioned close to thepatient, the tower 30 may be stowed in a remote location to stay out ofthe way during a procedure.

In support of the robotic systems described above, the tower 30 mayinclude component(s) of a computer-based control system that storescomputer program instructions, for example, within a non-transitorycomputer-readable storage medium such as a persistent magnetic storagedrive, solid state drive, etc. The execution of those instructions,whether the execution occurs in the tower 30 or the cart 11, may controlthe entire system or sub-system(s) thereof. For example, when executedby a processor of the computer system, the instructions may cause thecomponents of the robotics system to actuate the relevant carriages andarm mounts, actuate the robotics arms, and control the medicalinstruments. For example, in response to receiving the control signal,the motors in the joints of the robotics arms may position the arms intoa certain posture.

The tower 30 may also include a pump, flow meter, valve control, and/orfluid access in order to provide controlled irrigation and aspirationcapabilities to the system that may be deployed through the endoscope13. These components may also be controlled using the computer system ofthe tower 30. In some embodiments, irrigation and aspirationcapabilities may be delivered directly to the endoscope 13 throughseparate cable(s).

The tower 30 may include a voltage and surge protector designed toprovide filtered and protected electrical power to the cart 11, therebyavoiding placement of a power transformer and other auxiliary powercomponents in the cart 11, resulting in a smaller, more moveable cart11.

The tower 30 may also include support equipment for the sensors deployedthroughout the robotic system 10. For example, the tower 30 may includeoptoelectronics equipment for detecting, receiving, and processing datareceived from the optical sensors or cameras throughout the roboticsystem 10. In combination with the control system, such optoelectronicsequipment may be used to generate real-time images for display in anynumber of consoles deployed throughout the system, including in thetower 30. Similarly, the tower 30 may also include an electronicsubsystem for receiving and processing signals received from deployedelectromagnetic (EM) sensors. The tower 30 may also be used to house andposition an EM field generator for detection by EM sensors in or on themedical instrument.

The tower 30 may also include a console 31 in addition to other consolesavailable in the rest of the system, e.g., console mounted on top of thecart. The console 31 may include a user interface and a display screen,such as a touchscreen, for the physician operator. Consoles in thesystem 10 are generally designed to provide both robotic controls aswell as preoperative and real-time information of the procedure, such asnavigational and localization information of the endoscope 13. When theconsole 31 is not the only console available to the physician, it may beused by a second operator, such as a nurse, to monitor the health orvitals of the patient and the operation of the system 10, as well as toprovide procedure-specific data, such as navigational and localizationinformation. In other embodiments, the console 31 is housed in a bodythat is separate from the tower 30.

The tower 30 may be coupled to the cart 11 and endoscope 13 through oneor more cables or connections (not shown). In some embodiments, thesupport functionality from the tower 30 may be provided through a singlecable to the cart 11, simplifying and de-cluttering the operating room.In other embodiments, specific functionality may be coupled in separatecabling and connections. For example, while power may be providedthrough a single power cable to the cart 11, the support for controls,optics, fluidics, and/or navigation may be provided through a separatecable.

FIG. 2 provides a detailed illustration of an embodiment of the cart 11from the cart-based robotically-enabled system shown in FIG. 1. The cart11 generally includes an elongated support structure 14 (often referredto as a “column”), a cart base 15, and a console 16 at the top of thecolumn 14. The column 14 may include one or more carriages, such as acarriage 17 (alternatively “arm support”) for supporting the deploymentof one or more robotic arms 12 (three shown in FIG. 2). The carriage 17may include individually configurable arm mounts that rotate along aperpendicular axis to adjust the base of the robotic arms 12 for betterpositioning relative to the patient. The carriage 17 also includes acarriage interface 19 that allows the carriage 17 to verticallytranslate along the column 14.

The carriage interface 19 is connected to the column 14 through slots,such as slot 20, that are positioned on opposite sides of the column 14to guide the vertical translation of the carriage 17. The slot 20contains a vertical translation interface to position and hold thecarriage 17 at various vertical heights relative to the cart base 15.Vertical translation of the carriage 17 allows the cart 11 to adjust thereach of the robotic arms 12 to meet a variety of table heights, patientsizes, and physician preferences. Similarly, the individuallyconfigurable arm mounts on the carriage 17 allow the robotic arm base 21of the robotic arms 12 to be angled in a variety of configurations.

In some embodiments, the slot 20 may be supplemented with slot coversthat are flush and parallel to the slot surface to prevent dirt andfluid ingress into the internal chambers of the column 14 and thevertical translation interface as the carriage 17 vertically translates.The slot covers may be deployed through pairs of spring spoolspositioned near the vertical top and bottom of the slot 20. The coversare coiled within the spools until deployed to extend and retract fromtheir coiled state as the carriage 17 vertically translates up and down.The spring-loading of the spools provides force to retract the coverinto a spool when the carriage 17 translates towards the spool, whilealso maintaining a tight seal when the carriage 17 translates away fromthe spool. The covers may be connected to the carriage 17 using, forexample, brackets in the carriage interface 19 to ensure properextension and retraction of the cover as the carriage 17 translates.

The column 14 may internally comprise mechanisms, such as gears andmotors, that are designed to use a vertically aligned lead screw totranslate the carriage 17 in a mechanized fashion in response to controlsignals generated in response to user inputs, e.g., inputs from theconsole 16.

The robotic arms 12 may generally comprise robotic arm bases 21 and endeffectors 22, separated by a series of linkages 23 that are connected bya series of joints 24, each joint comprising an independent actuator,each actuator comprising an independently controllable motor. Eachindependently controllable joint represents an independent degree offreedom available to the robotic arm 12. Each of the robotic arms 12 mayhave seven joints, and thus provide seven degrees of freedom. Amultitude of joints result in a multitude of degrees of freedom,allowing for “redundant” degrees of freedom. Having redundant degrees offreedom allows the robotic arms 12 to position their respective endeffectors 22 at a specific position, orientation, and trajectory inspace using different linkage positions and joint angles. This allowsfor the system to position and direct a medical instrument from adesired point in space while allowing the physician to move the armjoints into a clinically advantageous position away from the patient tocreate greater access, while avoiding arm collisions.

The cart base 15 balances the weight of the column 14, carriage 17, androbotic arms 12 over the floor. Accordingly, the cart base 15 housesheavier components, such as electronics, motors, power supply, as wellas components that either enable movement and/or immobilize the cart 11.For example, the cart base 15 includes rollable wheel-shaped casters 25that allow for the cart 11 to easily move around the room prior to aprocedure. After reaching the appropriate position, the casters 25 maybe immobilized using wheel locks to hold the cart 11 in place during theprocedure.

Positioned at the vertical end of the column 14, the console 16 allowsfor both a user interface for receiving user input and a display screen(or a dual-purpose device such as, for example, a touchscreen 26) toprovide the physician user with both preoperative and intraoperativedata. Potential preoperative data on the touchscreen 26 may includepreoperative plans, navigation and mapping data derived frompreoperative computerized tomography (CT) scans, and/or notes frompreoperative patient interviews. Intraoperative data on display mayinclude optical information provided from the tool, sensor andcoordinate information from sensors, as well as vital patientstatistics, such as respiration, heart rate, and/or pulse. The console16 may be positioned and tilted to allow a physician to access theconsole 16 from the side of the column 14 opposite the carriage 17. Fromthis position, the physician may view the console 16, robotic arms 12,and patient while operating the console 16 from behind the cart 11. Asshown, the console 16 also includes a handle 27 to assist withmaneuvering and stabilizing the cart 11.

FIG. 3 illustrates an embodiment of a robotically-enabled system 10arranged for ureteroscopy. In a ureteroscopic procedure, the cart 11 maybe positioned to deliver a ureteroscope 32, a procedure-specificendoscope designed to traverse a patient's urethra and ureter, to thelower abdominal area of the patient. In a ureteroscopy, it may bedesirable for the ureteroscope 32 to be directly aligned with thepatient's urethra to reduce friction and forces on the sensitive anatomyin the area. As shown, the cart 11 may be aligned at the foot of thetable to allow the robotic arms 12 to position the ureteroscope 32 fordirect linear access to the patient's urethra. From the foot of thetable, the robotic arms 12 may insert the ureteroscope 32 along thevirtual rail 33 directly into the patient's lower abdomen through theurethra.

After insertion into the urethra, using similar control techniques as inbronchoscopy, the ureteroscope 32 may be navigated into the bladder,ureters, and/or kidneys for diagnostic and/or therapeutic applications.For example, the ureteroscope 32 may be directed into the ureter andkidneys to break up kidney stone build up using a laser or ultrasoniclithotripsy device deployed down the working channel of the ureteroscope32. After lithotripsy is complete, the resulting stone fragments may beremoved using baskets deployed down the ureteroscope 32.

FIG. 4 illustrates an embodiment of a robotically-enabled system 10similarly arranged for a vascular procedure. In a vascular procedure,the system 10 may be configured such that the cart 11 may deliver amedical instrument 34, such as a steerable catheter, to an access pointin the femoral artery in the patient's leg. The femoral artery presentsboth a larger diameter for navigation as well as a relatively lesscircuitous and tortuous path to the patient's heart, which simplifiesnavigation. As in a ureteroscopic procedure, the cart 11 may bepositioned towards the patient's legs and lower abdomen to allow therobotic arms 12 to provide a virtual rail 35 with direct linear accessto the femoral artery access point in the patient's thigh/hip region.After insertion into the artery, the medical instrument 34 may bedirected and inserted by translating the instrument drivers 28.Alternatively, the cart may be positioned around the patient's upperabdomen in order to reach alternative vascular access points, such as,for example, the carotid and brachial arteries near the shoulder andwrist.

B. Robotic System—Table.

Embodiments of the robotically-enabled medical system may alsoincorporate the patient's table. Incorporation of the table reduces theamount of capital equipment within the operating room by removing thecart, which allows greater access to the patient. FIG. 5 illustrates anembodiment of such a robotically-enabled system arranged for abronchoscopic procedure. System 36 includes a support structure orcolumn 37 for supporting platform 38 (shown as a “table” or “bed”) overthe floor. Much like in the cart-based systems, the end effectors of therobotic arms 39 of the system 36 comprise instrument drivers 42 that aredesigned to manipulate an elongated medical instrument, such as abronchoscope 40 in FIG. 5, through or along a virtual rail 41 formedfrom the linear alignment of the instrument drivers 42. In practice, aC-arm for providing fluoroscopic imaging may be positioned over thepatient's upper abdominal area by placing the emitter and detectoraround the table 38.

FIG. 6 provides an alternative view of the system 36 without the patientand medical instrument for discussion purposes. As shown, the column 37may include one or more carriages 43 shown as ring-shaped in the system36, from which the one or more robotic arms 39 may be based. Thecarriages 43 may translate along a vertical column interface 44 thatruns the length of the column 37 to provide different vantage pointsfrom which the robotic arms 39 may be positioned to reach the patient.The carriage(s) 43 may rotate around the column 37 using a mechanicalmotor positioned within the column 37 to allow the robotic arms 39 tohave access to multiples sides of the table 38, such as, for example,both sides of the patient. In embodiments with multiple carriages, thecarriages may be individually positioned on the column and may translateand/or rotate independently of the other carriages. While the carriages43 need not surround the column 37 or even be circular, the ring-shapeas shown facilitates rotation of the carriages 43 around the column 37while maintaining structural balance. Rotation and translation of thecarriages 43 allows the system 36 to align the medical instruments, suchas endoscopes and laparoscopes, into different access points on thepatient. In other embodiments (not shown), the system 36 can include apatient table or bed with adjustable arm supports in the form of bars orrails extending alongside it. One or more robotic arms 39 (e.g., via ashoulder with an elbow joint) can be attached to the adjustable armsupports, which can be vertically adjusted. By providing verticaladjustment, the robotic arms 39 are advantageously capable of beingstowed compactly beneath the patient table or bed, and subsequentlyraised during a procedure.

The robotic arms 39 may be mounted on the carriages 43 through a set ofarm mounts 45 comprising a series of joints that may individually rotateand/or telescopically extend to provide additional configurability tothe robotic arms 39. Additionally, the arm mounts 45 may be positionedon the carriages 43 such that, when the carriages 43 are appropriatelyrotated, the arm mounts 45 may be positioned on either the same side ofthe table 38 (as shown in FIG. 6), on opposite sides of the table 38 (asshown in FIG. 9), or on adjacent sides of the table 38 (not shown).

The column 37 structurally provides support for the table 38, and a pathfor vertical translation of the carriages 43. Internally, the column 37may be equipped with lead screws for guiding vertical translation of thecarriages, and motors to mechanize the translation of the carriages 43based the lead screws. The column 37 may also convey power and controlsignals to the carriages 43 and the robotic arms 39 mounted thereon.

The table base 46 serves a similar function as the cart base 15 in thecart 11 shown in FIG. 2, housing heavier components to balance thetable/bed 38, the column 37, the carriages 43, and the robotic arms 39.The table base 46 may also incorporate rigid casters to providestability during procedures. Deployed from the bottom of the table base46, the casters may extend in opposite directions on both sides of thebase 46 and retract when the system 36 needs to be moved.

With continued reference to FIG. 6, the system 36 may also include atower (not shown) that divides the functionality of the system 36between the table and the tower to reduce the form factor and bulk ofthe table. As in earlier disclosed embodiments, the tower may provide avariety of support functionalities to the table, such as processing,computing, and control capabilities, power, fluidics, and/or optical andsensor processing. The tower may also be movable to be positioned awayfrom the patient to improve physician access and de-clutter theoperating room. Additionally, placing components in the tower allows formore storage space in the table base 46 for potential stowage of therobotic arms 39. The tower may also include a master controller orconsole that provides both a user interface for user input, such askeyboard and/or pendant, as well as a display screen (or touchscreen)for preoperative and intraoperative information, such as real-timeimaging, navigation, and tracking information. In some embodiments, thetower may also contain holders for gas tanks to be used forinsufflation.

In some embodiments, a table base may stow and store the robotic armswhen not in use. FIG. 7 illustrates a system 47 that stows robotic armsin an embodiment of the table-based system. In the system 47, carriages48 may be vertically translated into base 49 to stow robotic arms 50,arm mounts 51, and the carriages 48 within the base 49. Base covers 52may be translated and retracted open to deploy the carriages 48, armmounts 51, and robotic arms 50 around column 53, and closed to stow toprotect them when not in use. The base covers 52 may be sealed with amembrane 54 along the edges of its opening to prevent dirt and fluidingress when closed.

FIG. 8 illustrates an embodiment of a robotically-enabled table-basedsystem configured for a ureteroscopic procedure. In a ureteroscopy, thetable 38 may include a swivel portion 55 for positioning a patientoff-angle from the column 37 and table base 46. The swivel portion 55may rotate or pivot around a pivot point (e.g., located below thepatient's head) in order to position the bottom portion of the swivelportion 55 away from the column 37. For example, the pivoting of theswivel portion 55 allows a C-arm (not shown) to be positioned over thepatient's lower abdomen without competing for space with the column (notshown) below table 38. By rotating the carriage (not shown) around thecolumn 37, the robotic arms 39 may directly insert a ureteroscope 56along a virtual rail 57 into the patient's groin area to reach theurethra. In a ureteroscopy, stirrups 58 may also be fixed to the swivelportion 55 of the table 38 to support the position of the patient's legsduring the procedure and allow clear access to the patient's groin area.

In a laparoscopic procedure, through small incision(s) in the patient'sabdominal wall, minimally invasive instruments may be inserted into thepatient's anatomy. In some embodiments, the minimally invasiveinstruments comprise an elongated rigid member, such as a shaft, whichis used to access anatomy within the patient. After inflation of thepatient's abdominal cavity, the instruments may be directed to performsurgical or medical tasks, such as grasping, cutting, ablating,suturing, etc. In some embodiments, the instruments can comprise ascope, such as a laparoscope. FIG. 9 illustrates an embodiment of arobotically-enabled table-based system configured for a laparoscopicprocedure. As shown in FIG. 9, the carriages 43 of the system 36 may berotated and vertically adjusted to position pairs of the robotic arms 39on opposite sides of the table 38, such that instrument 59 may bepositioned using the arm mounts 45 to be passed through minimalincisions on both sides of the patient to reach his/her abdominalcavity.

To accommodate laparoscopic procedures, the robotically-enabled tablesystem may also tilt the platform to a desired angle. FIG. 10illustrates an embodiment of the robotically-enabled medical system withpitch or tilt adjustment. As shown in FIG. 10, the system 36 mayaccommodate tilt of the table 38 to position one portion of the table ata greater distance from the floor than the other. Additionally, the armmounts 45 may rotate to match the tilt such that the robotic arms 39maintain the same planar relationship with the table 38. To accommodatesteeper angles, the column 37 may also include telescoping portions 60that allow vertical extension of the column 37 to keep the table 38 fromtouching the floor or colliding with the table base 46.

FIG. 11 provides a detailed illustration of the interface between thetable 38 and the column 37. Pitch rotation mechanism 61 may beconfigured to alter the pitch angle of the table 38 relative to thecolumn 37 in multiple degrees of freedom. The pitch rotation mechanism61 may be enabled by the positioning of orthogonal axes 1, 2 at thecolumn-table interface, each axis actuated by a separate motor 3, 4responsive to an electrical pitch angle command. Rotation along onescrew 5 would enable tilt adjustments in one axis 1, while rotationalong the other screw 6 would enable tilt adjustments along the otheraxis 2. In some embodiments, a ball joint can be used to alter the pitchangle of the table 38 relative to the column 37 in multiple degrees offreedom.

For example, pitch adjustments are particularly useful when trying toposition the table in a Trendelenburg position, i.e., position thepatient's lower abdomen at a higher position from the floor than thepatient's upper abdomen, for lower abdominal surgery. The Trendelenburgposition causes the patient's internal organs to slide towards his/herupper abdomen through the force of gravity, clearing out the abdominalcavity for minimally invasive tools to enter and perform lower abdominalsurgical or medical procedures, such as laparoscopic prostatectomy.

FIGS. 12 and 13 illustrate isometric and end views of an alternativeembodiment of a table-based surgical robotics system 100. The surgicalrobotics system 100 includes one or more adjustable arm supports 105that can be configured to support one or more robotic arms (see, forexample, FIG. 14) relative to a table 101. In the illustratedembodiment, a single adjustable arm support 105 is shown, though anadditional arm support 105 can be provided on an opposite side of thetable 101. The adjustable arm support 105 can be configured so that itcan move relative to the table 101 to adjust and/or vary the position ofthe adjustable arm support 105 and/or any robotic arms mounted theretorelative to the table 101. For example, the adjustable arm support 105may be adjusted one or more degrees of freedom relative to the table101. The adjustable arm support 105 provides high versatility to thesystem 100, including the ability to easily stow the one or moreadjustable arm supports 105 and any robotics arms attached theretobeneath the table 101. The adjustable arm support 105 can be elevatedfrom the stowed position to a position below an upper surface of thetable 101. In other embodiments, the adjustable arm support 105 can beelevated from the stowed position to a position above an upper surfaceof the table 101.

The adjustable arm support 105 can provide several degrees of freedom,including lift, lateral translation, tilt, etc. In the illustratedembodiment of FIGS. 12 and 13, the arm support 105 is configured withfour degrees of freedom, which are illustrated with arrows in FIG. 12. Afirst degree of freedom allows for adjustment of the adjustable armsupport 105 in the z-direction (“Z-lift”). For example, the adjustablearm support 105 can include a carriage 109 configured to move up or downalong or relative to a column 102 supporting the table 101. A seconddegree of freedom can allow the adjustable arm support 105 to tilt. Forexample, the adjustable arm support 105 can include a rotary joint,which can allow the adjustable arm support 105 to be aligned with thebed in a Trendelenburg position. A third degree of freedom can allow theadjustable arm support 105 to “pivot up,” which can be used to adjust adistance between a side of the table 101 and the adjustable arm support105. A fourth degree of freedom can permit translation of the adjustablearm support 105 along a longitudinal length of the table.

The surgical robotics system 100 in FIGS. 12 and 13 can comprise a tablesupported by a column 102 that is mounted to a base 103. The base 103and the column 102 support the table 101 relative to a support surface.A floor axis 131 and a support axis 133 are shown in FIG. 13.

The adjustable arm support 105 can be mounted to the column 102. Inother embodiments, the arm support 105 can be mounted to the table 101or base 103. The adjustable arm support 105 can include a carriage 109,a bar or rail connector 111 and a bar or rail 107. In some embodiments,one or more robotic arms mounted to the rail 107 can translate and moverelative to one another.

The carriage 109 can be attached to the column 102 by a first joint 113,which allows the carriage 109 to move relative to the column 102 (e.g.,such as up and down a first or vertical axis 123). The first joint 113can provide the first degree of freedom (Z-lift) to the adjustable armsupport 105. The adjustable arm support 105 can include a second joint115, which provides the second degree of freedom (tilt) for theadjustable arm support 105. The adjustable arm support 105 can include athird joint 117, which can provide the third degree of freedom (“pivotup”) for the adjustable arm support 105. An additional joint 119 (shownin FIG. 13) can be provided that mechanically constrains the third joint117 to maintain an orientation of the rail 107 as the rail connector 111is rotated about a third axis 127. The adjustable arm support 105 caninclude a fourth joint 121, which can provide a fourth degree of freedom(translation) for the adjustable arm support 105 along a fourth axis129.

FIG. 14 illustrates an end view of the surgical robotics system 140Awith two adjustable arm supports 105A, 105B mounted on opposite sides ofa table 101. A first robotic arm 142A is attached to the bar or rail107A of the first adjustable arm support 105B. The first robotic arm142A includes a base 144A attached to the rail 107A. The distal end ofthe first robotic arm 142A includes an instrument drive mechanism 146Athat can attach to one or more robotic medical instruments or tools.Similarly, the second robotic arm 142B includes a base 144B attached tothe rail 107B. The distal end of the second robotic arm 142B includes aninstrument drive mechanism 146B. The instrument drive mechanism 146B canbe configured to attach to one or more robotic medical instruments ortools.

In some embodiments, one or more of the robotic arms 142A, 142Bcomprises an arm with seven or more degrees of freedom. In someembodiments, one or more of the robotic arms 142A, 142B can includeeight degrees of freedom, including an insertion axis (1-degree offreedom including insertion), a wrist (3-degrees of freedom includingwrist pitch, yaw and roll), an elbow (1-degree of freedom includingelbow pitch), a shoulder (2-degrees of freedom including shoulder pitchand yaw), and base 144A, 144B (1-degree of freedom includingtranslation). In some embodiments, the insertion degree of freedom canbe provided by the robotic arm 142A, 142B, while in other embodiments,the instrument itself provides insertion via an instrument-basedinsertion architecture.

C. Instrument Driver & Interface.

The end effectors of the system's robotic arms may comprise (i) aninstrument driver (alternatively referred to as “instrument drivemechanism” or “instrument device manipulator”) that incorporateselectro-mechanical means for actuating the medical instrument and (ii) aremovable or detachable medical instrument, which may be devoid of anyelectro-mechanical components, such as motors. This dichotomy may bedriven by the need to sterilize medical instruments used in medicalprocedures, and the inability to adequately sterilize expensive capitalequipment due to their intricate mechanical assemblies and sensitiveelectronics. Accordingly, the medical instruments may be designed to bedetached, removed, and interchanged from the instrument driver (and thusthe system) for individual sterilization or disposal by the physician orthe physician's staff. In contrast, the instrument drivers need not bechanged or sterilized, and may be draped for protection.

FIG. 15 illustrates an example instrument driver. Positioned at thedistal end of a robotic arm, instrument driver 62 comprises one or moredrive units 63 arranged with parallel axes to provide controlled torqueto a medical instrument via drive shafts 64. Each drive unit 63comprises an individual drive shaft 64 for interacting with theinstrument, a gear head 65 for converting the motor shaft rotation to adesired torque, a motor 66 for generating the drive torque, an encoder67 to measure the speed of the motor shaft and provide feedback tocontrol circuitry, and control circuitry 68 for receiving controlsignals and actuating the drive unit. Each drive unit 63 beingindependently controlled and motorized, the instrument driver 62 mayprovide multiple (e.g., four as shown in FIG. 15) independent driveoutputs to the medical instrument. In operation, the control circuitry68 would receive a control signal, transmit a motor signal to the motor66, compare the resulting motor speed as measured by the encoder 67 withthe desired speed, and modulate the motor signal to generate the desiredtorque.

For procedures that require a sterile environment, the robotic systemmay incorporate a drive interface, such as a sterile adapter connectedto a sterile drape, that sits between the instrument driver and themedical instrument. The chief purpose of the sterile adapter is totransfer angular motion from the drive shafts of the instrument driverto the drive inputs of the instrument while maintaining physicalseparation, and thus sterility, between the drive shafts and driveinputs. Accordingly, an example sterile adapter may comprise a series ofrotational inputs and outputs intended to be mated with the drive shaftsof the instrument driver and drive inputs on the instrument. Connectedto the sterile adapter, the sterile drape, comprised of a thin, flexiblematerial such as transparent or translucent plastic, is designed tocover the capital equipment, such as the instrument driver, robotic arm,and cart (in a cart-based system) or table (in a table-based system).Use of the drape would allow the capital equipment to be positionedproximate to the patient while still being located in an area notrequiring sterilization (i.e., non-sterile field). On the other side ofthe sterile drape, the medical instrument may interface with the patientin an area requiring sterilization (i.e., sterile field).

D. Medical Instrument.

FIG. 16 illustrates an example medical instrument with a pairedinstrument driver. Like other instruments designed for use with arobotic system, medical instrument 70 comprises an elongated shaft 71(or elongate body) and an instrument base 72. The instrument base 72,also referred to as an “instrument handle” due to its intended designfor manual interaction by the physician, may generally compriserotatable drive inputs 73, e.g., receptacles, pulleys or spools, thatare designed to be mated with drive outputs 74 that extend through adrive interface on instrument driver 75 at the distal end of robotic arm76. When physically connected, latched, and/or coupled, the mated driveinputs 73 of the instrument base 72 may share axes of rotation with thedrive outputs 74 in the instrument driver 75 to allow the transfer oftorque from the drive outputs 74 to the drive inputs 73. In someembodiments, the drive outputs 74 may comprise splines that are designedto mate with receptacles on the drive inputs 73.

The elongated shaft 71 is designed to be delivered through either ananatomical opening or lumen, e.g., as in endoscopy, or a minimallyinvasive incision, e.g., as in laparoscopy. The elongated shaft 71 maybe either flexible (e.g., having properties similar to an endoscope) orrigid (e.g., having properties similar to a laparoscope) or contain acustomized combination of both flexible and rigid portions. Whendesigned for laparoscopy, the distal end of a rigid elongated shaft maybe connected to an end effector extending from a jointed wrist formedfrom a clevis with at least one degree of freedom and a surgical tool ormedical instrument, such as, for example, a grasper or scissors, thatmay be actuated based on force from the tendons as the drive inputsrotate in response to torque received from the drive outputs 74 of theinstrument driver 75. When designed for endoscopy, the distal end of aflexible elongated shaft may include a steerable or controllable bendingsection that may be articulated and bent based on torque received fromthe drive outputs 74 of the instrument driver 75.

Torque from the instrument driver 75 is transmitted down the elongatedshaft 71 using tendons along the elongated shaft 71. These individualtendons, such as pull wires, may be individually anchored to individualdrive inputs 73 within the instrument handle 72. From the instrumenthandle 72, the tendons are directed down one or more pull lumens alongthe elongated shaft 71 and anchored at the distal portion of theelongated shaft 71, or in the wrist at the distal portion of theelongated shaft. During a surgical procedure, such as a laparoscopic,endoscopic or hybrid procedure, these tendons may be coupled to adistally mounted end effector, such as a wrist, grasper, or scissor.Under such an arrangement, torque exerted on drive inputs 73 wouldtransfer tension to the tendon, thereby causing the end effector toactuate in some way. In some embodiments, during a surgical procedure,the tendon may cause a joint to rotate about an axis, thereby causingthe end effector to move in one direction or another. Alternatively, thetendon may be connected to one or more jaws of a grasper at the distalend of the elongated shaft 71, where tension from the tendon causes thegrasper to close.

In endoscopy, the tendons may be coupled to a bending or articulatingsection positioned along the elongated shaft 71 (e.g., at the distalend) via adhesive, control ring, or other mechanical fixation. Whenfixedly attached to the distal end of a bending section, torque exertedon the drive inputs 73 would be transmitted down the tendons, causingthe softer, bending section (sometimes referred to as the articulablesection or region) to bend or articulate. Along the non-bendingsections, it may be advantageous to spiral or helix the individual pulllumens that direct the individual tendons along (or inside) the walls ofthe endoscope shaft to balance the radial forces that result fromtension in the pull wires. The angle of the spiraling and/or spacingtherebetween may be altered or engineered for specific purposes, whereintighter spiraling exhibits lesser shaft compression under load forces,while lower amounts of spiraling results in greater shaft compressionunder load forces, but limits bending. On the other end of the spectrum,the pull lumens may be directed parallel to the longitudinal axis of theelongated shaft 71 to allow for controlled articulation in the desiredbending or articulable sections.

In endoscopy, the elongated shaft 71 houses a number of components toassist with the robotic procedure. The shaft 71 may comprise a workingchannel for deploying surgical tools (or medical instruments),irrigation, and/or aspiration to the operative region at the distal endof the shaft 71. The shaft 71 may also accommodate wires and/or opticalfibers to transfer signals to/from an optical assembly at the distaltip, which may include an optical camera. The shaft 71 may alsoaccommodate optical fibers to carry light from proximally-located lightsources, such as light emitting diodes, to the distal end of the shaft71.

At the distal end of the instrument 70, the distal tip may also comprisethe opening of a working channel for delivering tools for diagnosticand/or therapy, irrigation, and aspiration to an operative site. Thedistal tip may also include a port for a camera, such as a fiberscope ora digital camera, to capture images of an internal anatomical space.Relatedly, the distal tip may also include ports for light sources forilluminating the anatomical space when using the camera.

In the example of FIG. 16, the drive shaft axes, and thus the driveinput axes, are orthogonal to the axis of the elongated shaft 71. Thisarrangement, however, complicates roll capabilities for the elongatedshaft 71. Rolling the elongated shaft 71 along its axis while keepingthe drive inputs 73 static results in undesirable tangling of thetendons as they extend off the drive inputs 73 and enter pull lumenswithin the elongated shaft 71. The resulting entanglement of suchtendons may disrupt any control algorithms intended to predict movementof the flexible elongated shaft 71 during an endoscopic procedure.

FIG. 17 illustrates an alternative design for an instrument driver andinstrument where the axes of the drive units are parallel to the axis ofthe elongated shaft of the instrument. As shown, a circular instrumentdriver 80 comprises four drive units with their drive outputs 81 alignedin parallel at the end of a robotic arm 82. The drive units, and theirrespective drive outputs 81, are housed in a rotational assembly 83 ofthe instrument driver 80 that is driven by one of the drive units withinthe assembly 83. In response to torque provided by the rotational driveunit, the rotational assembly 83 rotates along a circular bearing thatconnects the rotational assembly 83 to the non-rotational portion 84 ofthe instrument driver 80. Power and controls signals may be communicatedfrom the non-rotational portion 84 of the instrument driver 80 to therotational assembly 83 through electrical contacts that may bemaintained through rotation by a brushed slip ring connection (notshown). In other embodiments, the rotational assembly 83 may beresponsive to a separate drive unit that is integrated into thenon-rotatable portion 84, and thus not in parallel to the other driveunits. The rotational mechanism 83 allows the instrument driver 80 torotate the drive units, and their respective drive outputs 81, as asingle unit around an instrument driver axis 85.

Like earlier disclosed embodiments, an instrument 86 may comprise anelongated shaft portion 88 and an instrument base 87 (shown with atransparent external skin for discussion purposes) comprising aplurality of drive inputs 89 (such as receptacles, pulleys, and spools)that are configured to receive the drive outputs 81 in the instrumentdriver 80. Unlike prior disclosed embodiments, the instrument shaft 88extends from the center of the instrument base 87 with an axissubstantially parallel to the axes of the drive inputs 89, rather thanorthogonal as in the design of FIG. 16.

When coupled to the rotational assembly 83 of the instrument driver 80,the medical instrument 86, comprising instrument base 87 and instrumentshaft 88, rotates in combination with the rotational assembly 83 aboutthe instrument driver axis 85. Since the instrument shaft 88 ispositioned at the center of instrument base 87, the instrument shaft 88is coaxial with instrument driver axis 85 when attached. Thus, rotationof the rotational assembly 83 causes the instrument shaft 88 to rotateabout its own longitudinal axis. Moreover, as the instrument base 87rotates with the instrument shaft 88, any tendons connected to the driveinputs 89 in the instrument base 87 are not tangled during rotation.Accordingly, the parallelism of the axes of the drive outputs 81, driveinputs 89, and instrument shaft 88 allows for the shaft rotation withouttangling any control tendons.

FIG. 18 illustrates an instrument 150 having an instrument basedinsertion architecture in accordance with some embodiments. Theinstrument 150 can be coupled to any of the instrument drivers discussedabove. The instrument 150 comprises an elongated shaft 152, an endeffector 162 connected to the shaft 152, and a handle 170 coupled to theshaft 152. The elongated shaft 152 comprises a tubular member having aproximal portion 154 and a distal portion 156. The elongated shaft 152comprises one or more channels or grooves 158 along its outer surface.The grooves 158 are configured to receive one or more wires or cables180 therethrough. One or more cables 180 thus run along an outer surfaceof the elongated shaft 152. In other embodiments, cables 180 can alsorun through the elongated shaft 152. Manipulation of the one or morecables 180 (e.g., via an instrument driver) results in actuation of theend effector 162.

The instrument handle 170, which may also be referred to as aninstrument base, may generally comprise an attachment interface 172having one or more mechanical inputs 174, e.g., receptacles, pulleys orspools, that are designed to be reciprocally mated with one or moretorque couplers on an attachment surface of an instrument driver.

In some embodiments, the instrument 150 comprises a series of pulleys orcables that enable the elongated shaft 152 to translate relative to thehandle 170. In other words, the instrument 150 itself comprises aninstrument-based insertion architecture that accommodates insertion ofthe instrument, thereby minimizing the reliance on a robot arm toprovide insertion of the instrument 150. In other embodiments, a roboticarm can be largely responsible for instrument insertion.

E. Controller.

Any of the robotic systems described herein can include an input deviceor controller for manipulating an instrument attached to a robotic arm.In some embodiments, the controller can be coupled (e.g.,communicatively, electronically, electrically, wirelessly and/ormechanically) with an instrument such that manipulation of thecontroller causes a corresponding manipulation of the instrument e.g.,via master slave control.

FIG. 19 is a perspective view of an embodiment of a controller 182. Inthe present embodiment, the controller 182 comprises a hybrid controllerthat can have both impedance and admittance control. In otherembodiments, the controller 182 can utilize just impedance or passivecontrol. In other embodiments, the controller 182 can utilize justadmittance control. By being a hybrid controller, the controller 182advantageously can have a lower perceived inertia while in use.

In the illustrated embodiment, the controller 182 is configured to allowmanipulation of two medical instruments, and includes two handles 184.Each of the handles 184 is connected to a gimbal 186. Each gimbal 186 isconnected to a positioning platform 188.

As shown in FIG. 19, each positioning platform 188 includes a selectivecompliance assembly robot arm (SCARA) 198 coupled to a column 194 by aprismatic joint 196. The prismatic joints 196 are configured totranslate along the column 194 (e.g., along rails 197) to allow each ofthe handles 184 to be translated in the z-direction, providing a firstdegree of freedom. The SCARA 198 is configured to allow motion of thehandle 184 in an x-y plane, providing two additional degrees of freedom.

In some embodiments, one or more load cells are positioned in thecontroller. For example, in some embodiments, a load cell (not shown) ispositioned in the body of each of the gimbals 186. By providing a loadcell, portions of the controller 182 are capable of operating underadmittance control, thereby advantageously reducing the perceivedinertia of the controller while in use. In some embodiments, thepositioning platform 188 is configured for admittance control, while thegimbal 186 is configured for impedance control. In other embodiments,the gimbal 186 is configured for admittance control, while thepositioning platform 188 is configured for impedance control.Accordingly, for some embodiments, the translational or positionaldegrees of freedom of the positioning platform 188 can rely onadmittance control, while the rotational degrees of freedom of thegimbal 186 rely on impedance control.

F. Navigation and Control.

Traditional endoscopy may involve the use of fluoroscopy (e.g., as maybe delivered through a C-arm) and other forms of radiation-based imagingmodalities to provide endoluminal guidance to an operator physician. Incontrast, the robotic systems contemplated by this disclosure canprovide for non-radiation-based navigational and localization means toreduce physician exposure to radiation and reduce the amount ofequipment within the operating room. As used herein, the term“localization” may refer to determining and/or monitoring the positionof objects in a reference coordinate system. Technologies such aspreoperative mapping, computer vision, real-time EM tracking, and robotcommand data may be used individually or in combination to achieve aradiation-free operating environment. In other cases, whereradiation-based imaging modalities are still used, the preoperativemapping, computer vision, real-time EM tracking, and robot command datamay be used individually or in combination to improve upon theinformation obtained solely through radiation-based imaging modalities.

FIG. 20 is a block diagram illustrating a localization system 90 thatestimates a location of one or more elements of the robotic system, suchas the location of the instrument, in accordance to an exampleembodiment. The localization system 90 may be a set of one or morecomputer devices configured to execute one or more instructions. Thecomputer devices may be embodied by a processor (or processors) andcomputer-readable memory in one or more components discussed above. Byway of example and not limitation, the computer devices may be in thetower 30 shown in FIG. 1, the cart 11 shown in FIGS. 1-4, the beds shownin FIGS. 5-14, etc.

As shown in FIG. 20, the localization system 90 may include alocalization module 95 that processes input data 91-94 to generatelocation data 96 for the distal tip of a medical instrument. Thelocation data 96 may be data or logic that represents a location and/ororientation of the distal end of the instrument relative to a frame ofreference. The frame of reference can be a frame of reference relativeto the anatomy of the patient or to a known object, such as an EM fieldgenerator (see discussion below for the EM field generator).

The various input data 91-94 are now described in greater detail.Preoperative mapping may be used by the localization module 95 togenerate model data 91. Preoperative mapping may be accomplished throughthe use of the collection of low dose CT scans. Preoperative CT scansare reconstructed into three-dimensional images, which are visualized,e.g. as “slices” of a cutaway view of the patient's internal anatomy.When analyzed in the aggregate, image-based models for anatomicalcavities, spaces and structures of the patient's anatomy, such as apatient lung network, may be generated. Techniques such as center-linegeometry may be determined and approximated from the CT images todevelop a three-dimensional volume of the patient's anatomy, referred toas model data 91 (also referred to as “preoperative model data” whengenerated using only preoperative CT scans). The use of center-linegeometry is discussed in U.S. patent application Ser. No. 14/523,760,the contents of which are herein incorporated in its entirety. Networktopological models may also be derived from the CT-images, and areparticularly appropriate for bronchoscopy.

In some embodiments, the instrument may be equipped with a camera toprovide vision data (or image data) 92 to the localization module 95.The localization module 95 may process the vision data 92 to enable oneor more vision-based (or image-based) location tracking modules orfeatures. For example, the preoperative model data 91 may be used inconjunction with the vision data 92 to enable computer vision-basedtracking of the medical instrument (e.g., an endoscope or an instrumentadvance through a working channel of the endoscope). For example, usingthe preoperative model data 91, the robotic system may generate alibrary of expected endoscopic images from the model based on theexpected path of travel of the endoscope, each image linked to alocation within the model. Intraoperatively, this library may bereferenced by the robotic system in order to compare real-time imagescaptured at the camera (e.g., a camera at a distal end of the endoscope)to those in the image library to assist localization.

Other computer vision-based tracking techniques use feature tracking todetermine motion of the camera, and thus the endoscope. Some features ofthe localization module 95 may identify circular geometries in thepreoperative model data 91 that correspond to anatomical lumens andtrack the change of those geometries to determine which anatomical lumenwas selected, as well as the relative rotational and/or translationalmotion of the camera. Use of a topological map may further enhancevision-based algorithms or techniques.

Optical flow, another computer vision-based technique, may analyze thedisplacement and translation of image pixels in a video sequence in thevision data 92 to infer camera movement. Examples of optical flowtechniques may include motion detection, object segmentationcalculations, luminance, motion compensated encoding, stereo disparitymeasurement, etc. Through the comparison of multiple frames overmultiple iterations, movement and location of the camera (and thus theendoscope) may be determined.

The localization module 95 may use real-time EM tracking and EM data 93to generate a real-time location of the endoscope in a global coordinatesystem that may be registered to the patient's anatomy, represented bythe preoperative model. In EM tracking, an EM sensor (or tracker)comprising one or more sensor coils embedded in one or more locationsand orientations in a medical instrument (e.g., an endoscopic tool)measures the variation in the EM field created by one or more static EMfield generators positioned at a known location. The locationinformation detected by the EM sensors is stored as EM data 93. The EMfield generator (or transmitter), may be placed close to the patient tocreate a low intensity magnetic field that the embedded sensor maydetect. The magnetic field induces small currents in the sensor coils ofthe EM sensor, which may be analyzed to determine the distance and anglebetween the EM sensor and the EM field generator. These distances andorientations may be intraoperatively “registered” to the patient anatomy(e.g., the preoperative model) in order to determine the geometrictransformation that aligns a single location in the coordinate systemwith a position in the preoperative model of the patient's anatomy. Onceregistered, an embedded EM tracker in one or more positions of themedical instrument (e.g., the distal tip of an endoscope) may providereal-time indications of the progression of the medical instrumentthrough the patient's anatomy.

Robotic command and kinematics data 94 may also be used by thelocalization module 95 to provide location data 96 for the roboticsystem. Device pitch and yaw resulting from articulation commands may bedetermined during preoperative calibration. Intraoperatively, thesecalibration measurements may be used in combination with known insertiondepth information to estimate the position of the instrument.Alternatively, these calculations may be analyzed in combination withEM, vision, and/or topological modeling to estimate the position of themedical instrument within the network.

As FIG. 20 shows, a number of other input data can be used by thelocalization module 95. For example, although not shown in FIG. 20, aninstrument utilizing shape-sensing fiber can provide shape data that thelocalization module 95 can use to determine the location and shape ofthe instrument.

The localization module 95 may use the input data 91-94 incombination(s). In some cases, such a combination may use aprobabilistic approach where the localization module 95 assigns aconfidence weight to the location determined from each of the input data91-94. Thus, where the EM data may not be reliable (as may be the casewhere there is EM interference) the confidence of the locationdetermined by the EM data 93 can be decrease and the localization module95 may rely more heavily on the vision data 92 and/or the roboticcommand and kinematics data 94.

As discussed above, the robotic systems discussed herein may be designedto incorporate a combination of one or more of the technologies above.The robotic system's computer-based control system, based in the tower,bed and/or cart, may store computer program instructions, for example,within a non-transitory computer-readable storage medium such as apersistent magnetic storage drive, solid state drive, or the like, that,upon execution, cause the system to receive and analyze sensor data anduser commands, generate control signals throughout the system, anddisplay the navigational and localization data, such as the position ofthe instrument within the global coordinate system, anatomical map, etc.

2. Robotically Controllable Field Generators

Embodiments of robotically controllable EM field generators that may beconfigured for use with robotic medical systems, such as those describedabove and others, are described herein. The robotically controllable EMfield generators can be configured to couple to (or otherwise beattached to or integrated into) a robotic arm of a robotic medicalsystem. Such robotically controllable EM field generators can provideadvantages over other EM field generators that are commonly used withrobotic medical systems, which usually require a more complexregistration step to relate a coordinate frame of the EM field to acoordinate frame of the robotic system or a global coordinate frame andgenerally cannot be repositioned during a procedure without having toredo the complex registration step.

By coupling an EM field generator to a robotic arm, the kinematics ofthe robotic arm can be used to register the coordinate frame of the EMfield to the robotic or global coordinate frame. Such registration basedon the kinematics of the robotic arm can occur automatically or withoutrequiring an operator to perform a separate registration step (such asthose registration steps described below with reference to FIGS. 22B and22C). This can eliminate the need for more complex or user involvedregistrations that are common with other types of EM field generatorsused with robotic medical systems. Further, with the EM field generatorcoupled to the robotic arm, the EM field generator becomes roboticallycontrollable or positionable. The robotic arm can be commanded totranslate, rotate, or a combination of both, to physically move the EMfield generator to various positions during a robotic medical procedure.

As will be described in more detail below, such robotically controllableEM field generators can provide several advantages and are useful in awide variety of applications. For example, use of such roboticallycontrollable EM field generators can eliminate the need to perform aseparate registration step to register the coordinate frame of the EMfield with the robotic or global coordinate frame, provide a wider fieldof view through the ability to move the EM field generator with therobotic arm, permit the use of smaller EM field generators (e.g.,compact field generators), simplify set up of the robotic medicalsystem, and increase the accuracy of the detected position of EMsensors, among others. Applications for such robotically controllable EMfield generators can include, for example, automatic tool tracking,automatic or enhanced field generator setup, EM focusing, anatomicalmapping, and others.

These and other features of the robotically controllable fieldgenerators will be described in more detail below after first providinga brief overview of the use of EM sensors and EM Field generatorsgenerally in robotic medical systems.

A. Brief Overview of Use of EM Sensors and EM Field Generators

Robotic medical systems may use various types of position sensors tofacilitate tracking and navigation of various tools and instruments. Asdescribed above, one type of position sensor that may be used in arobotic medical system is an EM position sensor (also referred to hereinas an EM sensor, beacon, or tracker). One or more EM sensors can beprovided, for example, on a portion of a tool or instrument used duringa medical procedure. The position(s) of the EM sensor(s) can bedetermined and used to track the position(s) of the correspondingtool(s) or instruments. One or more EM sensors can also be provided on apatient, for example, to track patient movement (such as movement due torespiration or other types of movement) during a procedure. As notedabove with respect to FIG. 20, a localization system 90 may use EM data93, along with various other types of data, to provide navigation andguidance information to an operator of the system.

EM sensors are used with one or more EM field generators that areconfigured to generate a low intensity magnetic field. When an EM sensoris positioned within the magnetic field generated by EM field generator(also referred to as the working volume of the EM field generator), theposition of the EM sensor relative to the EM field generator can bedetermined. For example, as described above with reference to FIG. 20,an EM sensor can comprise one or more sensor coils that measure thevariation in the magnetic field generated by an EM field generator. Themagnetic field can induce small currents in the sensor coils of the EMsensor that can be analyzed to determine, for example, the distance andangle between the EM sensor and the EM field generator.

In some embodiments, the position (e.g., the three-dimensional position)of the EM sensor relative to the EM field generator can be determined.In some embodiments, the orientation (e.g., the pitch, yaw, and/or roll)of the EM sensor can also be determined. A five degree of freedom (DoF)EM sensor can provide the three-dimensional position of the EM sensor aswell as the pitch and yaw of the EM sensor. A six DoF EM sensor canprovide the three-dimensional position of the EM sensor as well as thepitch, yaw, and roll of the EM sensor.

FIG. 21 illustrates an example of an EM field generator 202 and an EMsensor 204. The EM field generator 202 is configured to produce amagnetic field. In the illustrated embodiment, the magnetic field of theEM field generator 202 has a working volume 206. When the EM sensor 204is positioned within the working volume 206 of the magnetic fieldgenerated by the EM field generator 202 the position (or the positionand orientation) of the EM sensor 204 can be determined relative to anEM coordinate frame 208 associated with the EM field generator 202. Forexample, the distance and angle between the EM sensor 204 and an originof the EM coordinate frame 208 can be determined such that the position(e.g., the x, y, and z position) and/or the orientation (e.g., thepitch, yaw, and/or roll) of the EM sensor 204 within the working volume206 can be determined.

In the illustrated embodiment, the working volume 206 of EM fieldgenerator 202 is represented as having the shape of a rectangular prismprojecting from one side of the EM field generator 202. This, however,is merely representative of one type of working volume 206, and EM fieldgenerators 202 can be provided that produce working volumes 206 ofvarious sizes, shapes, and positions relative the EM field generator202.

FIG. 22A illustrates an example robotic medical system 200 that uses theEM field generator 202 of FIG. 21. The robotic medical system 200 may besimilar to the robotic medical systems described above with reference toFIGS. 1-20. In the illustrated embodiment, the robotic medical system200 comprises a cart 210 including two robotic arms 212. Althoughillustrated with two robotic arms 212, other numbers of robotic arms 212are also possible. For example, FIG. 2 (described above) illustrates anexample cart 11 that includes three robotic arms 12. Additionally, insome embodiments, the robotic arms 212 need not be attached to the cart210. In some embodiments, the robotic arms can be coupled to the bed orpatient platform 215, for example, as shown in FIGS. 5-10 and 14(described above). In some embodiments, the robotic medical system 200may comprise robotic arms 212 that are coupled to the cart 210, thepatient platform, and/or other components of the system.

The robotic arms 212 can be associated with a robotic coordinate frame216. Due to the known kinematics of the robotic arms 212, the positionof the robotic arms 212 (e.g., the positions of the distal ends of therobotic arms 212 and/or the positions of an instrument 214 attachedthereto) can be determined with reference to the robotic coordinateframe 216. For example, because the lengths of the various links thatmake up the robotic arms 212 are known and the angles of between thelinks of the robotic arms 212 can be determined, the positions of therobotic arms are kinematically defined within the robotic coordinateframe 216.

In some embodiments, the position and orientation of the cart 210relative to the patient platform 215 and patient can be determined, set,or controlled such that the robotic coordinate frame 216 can beconsidered a global coordinate frame that can be registered to thepatient or a portion of the patient's anatomy. For example, in someembodiments, up, down, right, left, etc., within the robotic coordinateframe 216 can correspond to up, down, right, left, etc., within theglobal coordinate frame that includes the patient, patient platform 215,and/or other components.

As shown in FIG. 22A, the tool or instrument 214 (e.g., a laparoscopicor endoscopic instrument) can be coupled to one of the robotic arms 212.The instrument 214 can be inserted into the patient to perform a medicalprocedure. Although only a single instrument 214 is illustrated, itshould be appreciated that other numbers of instruments 214 (e.g., twoor more instruments) can be used in other embodiments. One or morerobotic arms 212 are useable to position and control the instrument(s)214. An operator (not shown) can control the robotic arm(s) 212 and theinstrument 214 using a controller.

In the illustrated embodiment, the instrument 214 includes the EM sensor204 positioned at a distal end thereof. Further, the illustratedembodiment includes a second EM sensor 204 attached to the patient'schest, which can be used to track patient movement or motion, such asmotion caused by the patient's respiration.

FIG. 22A illustrates that, in some embodiments, the EM field generator202 can be positioned relative to the patient such that the workingvolume 206 of the field generator 202 overlaps with a portion of thepatient's anatomy at which the medical procedure is performed. As usedherein, a portion of the patient's anatomy at which a medical procedureis performed can be referred to as a medical site, and this site caninclude a site where any medical procedure can be performed, includingbiopsy, endoscopy, surgery, therapeutics, and the like. In someembodiments, the EM field generator 202 can be supported by a stand orother support structure (not shown) such that the working volume 206 ispositioned relative to the patient. In other embodiments, the EM fieldgenerator 202 can be supported by or attached to the patient platform215 or integrated into or supported by another component within theoperating environment (such as the cart 30 described above withreference to FIGS. 1, 3, and 4).

As described above with reference to FIG. 21, when the EM sensors 204are positioned within the working volume 206 of the EM field generator202, the positions of the EM sensors 204 within the EM coordinate frame208 can be determined. However, determining the positions of the EMsensors 204 within the EM coordinate frame 208 may not be particularlyuseful, unless the EM coordinate frame 208 has been registered to therobotic coordinate frame 216 or a global coordinate frame (which asnoted above can be the robotic coordinate frame 216). Thus, aregistration step, which relates the EM coordinate frame 208 to therobotic coordinate frame 216, is often required.

In some robotic medical systems, such as the illustrated robotic medicalsystem 200 of FIG. 22A, registering the EM coordinate frame 208 to therobotic coordinate frame 216 can require the operator to perform certainsteps to achieve the registration. The registration process can addsteps to a procedure and can increase procedure times. FIGS. 22B and 22Cillustrate example registration steps or procedure that can be used toregister the robotic coordinate frame 216 associated with the roboticmedical system 200 with the EM coordinate system 208 associated with theEM field generator 202. As will be described in more detail below, theneed to perform these types of registration steps or procedures can beeliminated or reduced by providing a robotic medical system with an EMfield generator that is attached to a robotic arm thereof, as shown, forexample, in FIG. 23.

FIG. 22B illustrates an example registration procedure that can be usedto register the robotic coordinate frame 216 associated with the roboticmedical system 200 with the EM coordinate system 208 associated with theEM field generator 202. To achieve registration, a plurality of pointsare identified within both the robotic coordinate frame 216 and the EMcoordinate frame 208. Once the plurality of points are identified withineach of the robotic coordinate frame 216 and the EM coordinate frame208, a registration algorithm can determine a transformation between thetwo coordinate frames. The registration procedure of FIG. 22B uses oneof the robotic arms 212 of the robotic system, the EM field generator202, an EM probe 203, and a registration fixture 205. In the illustratedembodiment, the registration fixture 205 comprises a cube havingfiducials or markers located at the vertices of the cube. Accordingly,in the illustrated embodiment, the registration fixture 205 compriseseight fiducials. Other numbers of fiducials and other shapes for theregistration fixture 205 may also be used. With the registration fixture205 positioned in a stationary location, the operator can command therobotic arm 212 to touch each of the fiducials of the registrationfixture 205. The position of each fiducial within the robotic coordinateframe 216 can thus be determined. The operator can then touch each ofthe fiducials with the EM probe 203. The EM probe 203 can include ahandheld EM sensor 204 having a well-defined tip position with respectto the field generator 202. By touching each of the fiducials with theEM probe 203, the position of each fiducial within the EM coordinateframe 208 can be determined. With the same points (the fiducials of theregistration fixture 205) now known within each of the roboticcoordinate frame 216 and the EM coordinate frame 208, a registrationalgorithm can be determine a transformation between the two coordinateframes.

FIG. 22C illustrates another example registration procedure that can beused to register the robotic coordinate frame 216 associated with therobotic medical system 200 with the EM coordinate system 208 associatedwith the EM field generator 202. This registration procedure uses one ofthe robotic arms 212 of the robotic system, the EM field generator 202,and an EM sensor 204 that is attached to the end effector of the roboticarm 212. With the EM sensor 204 that is attached to the end effector ofthe robotic arm 212, the operator can than command the robotic arm 212to move through or trace a registration trajectory 207. In theillustrated embodiment, the registration trajectory 207 comprises theshape of a cube, although other registration trajectories comprisingdifferent shapes can be used. At various points (e.g., the vertices ofthe cube shape) along the registration trajectory 207, the position ofthe end effector in the robotic coordinate frame 216 and the position ofthe EM sensor 204 in the EM coordinate frame 208 can be recorded. Once asufficient plurality of points are identified with both the roboticcoordinate frame 216 and the EM coordinate frame 208, a registrationalgorithm can be determine a transformation between the two coordinateframes.

In robotic systems such as the system 200 illustrated in FIG. 22A, aregistration procedure, such as either of the procedures described abovewith reference to FIGS. 22B and 22C, may be performed prior to beginningthe procedure to register the EM coordinate frame 208 and the roboticcoordinate frame 216.

Various disadvantages are associated with these types of registrations.For example, such registration can be tedious and time consuming. Suchregistrations are generally not considered automatic as they may requireuser input and/or may need to be performed prior to beginning a roboticmedical procedure. Further, such registrations can introduceinaccuracies into the system, for example, if the operator does notnavigate exactly to the fiducials. Additionally, such registrationsrequire that the position of the EM field generator 202 remains fixedduring the procedure. If the EM field generator 202 is moved (e.g.,accidently bumped by someone in the operating room or moved to allowaccess to the patient), the registration would need to be performedagain to reestablish the relationship. This can be problematic as the EMfield generator 202 may need to be moved to allow access to afluoroscopic C-arm, for example. In this case, the EM field generator202 would be removed, the C-arm would be brought into position tocapture one or more images, then the C-arm would be removed, and the EMfield generator 202 brought back into position, requiring the operatorto redo the EM coordinate frame to robotic or global coordinate frameregistration steps.

As will be described in more detail below, the need for these types ofregistrations can be reduced or eliminated through the use ofrobotically-controllable field generators that can be coupled directlyto the robotic arms, for example, as shown in FIGS. 23-25. By couplingthe robotically-controllable field generator to the robotic arm, therelationship or registration between the robotically-controllable fieldgenerator and the robotic arm can be kinematically determined, thusplacing the EM coordinate frame 208 within the robotic coordinate frame216.

Additionally, since the position of the EM field generator 202 mustremain fixed during the procedure in previous systems that useregistration procedures such as those described with reference to FIGS.22B and 22C, it may be necessary to use a physically larger, andsometimes more cumbersome, EM field generator 202 in order to produce aworking volume 206 with a sufficiently large size to cover the relevantmedical site. In some situations or applications, this may increase thedifficulty in setting up and positioning the EM field generator 202relative to the patient. In some situations or applications, this mayrestrict the available space around the patient during the procedureand/or require increased set up times. The use ofrobotically-controllable field generators that can be coupled directlyto the robotic arms, for example, as shown in FIGS. 22-25, can reduce oreliminate one or more of these restrictions associated with the EM fieldgenerator 202 as shown in FIG. 22A.

B. Overview of Robotically Controllable Field Generators

FIG. 23 illustrates an embodiment of a robotic medical system 300 thatincludes an embodiment of a robotically controllable EM field generator302 that is coupled to a robotic arm 212. As will be discussed in moredetail below, coupling the EM field generator 302 to the robotic arm 212can provide several benefits including, for example, (1) allowing for amore accurate and/or simpler registration between an EM coordinate frame308 of the EM field generator 302 and the robotic coordinate frame 216based on the kinematics of the robotic arm 212, and/or (2) allowing theposition of the EM field generator 302 to be moved or readjusted usingthe robotic arm 212.

Similar to the robotic medical system 200 of FIG. 22A, in the roboticmedical system 300 of FIG. 23, the robotic arms 212 can be associatedwith a robotic coordinate frame 216. Due to the known kinematics of therobotic arms 212, the position of the robotic arms 212 can be determinedwith reference to the robotic coordinate frame 216. For example, becausethe lengths of the various links that make up the robotic arms 212 areknown and the angles of between the links of the robotic arms 212 can bedetermined, the positions of the robotic arms 212 are kinematicallydefined within the robotic coordinate frame 216. Additionally, for someembodiments of the robotic medical system 300, the position andorientation of the cart 210 relative to the patient platform 215 andpatient can be determined, set, or controlled such that the roboticcoordinate frame 216 can be considered a global coordinate frame thatcan be registered to the patient or a portion of the patient's anatomy.For example, in some embodiments, up, down, right, left, etc., withinthe robotic coordinate frame 216 can correspond to up, down, right,left, etc., within the global coordinate frame that includes thepatient, patient platform 215, and/or other components. Doing so canfacilitate the operator's ability to navigate within the patient'sanatomy. As a more specific example, during a bronchoscopic orureteroscopic procedure in which the robotic system 300 comprises a cartwith the robotic arms 212 extending therefrom, the cart can bepositioned parallel to the bed. The bed and the cart have a commongravity vector (i.e., up points to ceiling for both of them). Using thisinformation, the system can show the user which direction is up(anterior), or down (posterior) as they are driving the bronchoscope orureteroscope and provide spatial context. In the case of a urologicapplication, the user may need to identify if the ureteroscope is in aposterior calyx or anterior calyx during target selection forpercutaneous access. Since the orientation of the system is known, thesystem can provide anatomical context to facilitate this navigation.

As shown in FIG. 23, the EM field generator 302 can be coupled to one ofthe robotic arms 212. In some embodiments, the EM field generator 302 iscoupled or attached to a distal end of the robotic arm 212, although theEM field generator 302 may be coupled or attached to the robotic arm 212in other positions (e.g., positions between the distal and proximal endsof the arm). FIGS. 24 and 25, described below, illustrate a moredetailed embodiment of the EM field generator 302 and show how it can,in some embodiments, couple to the robotic arm 212. Other methods andmechanisms for coupling the EM field generator 302 to the robotic arm212 are also possible. Further, in some embodiments, the EM fieldgenerator 302 can be integrated into (e.g., can be a component on orwithin) the robotic arm 212 itself. For example, the EM field generator302 can be integrated into one of the links of the robotic arm 212 (suchas the distal most link) or into an instrument driver or instrumentdrive mechanism positioned, for example, at a distal end of the roboticarm 212. Example instrument drivers 62, 75, 83, 146 into which the EMfield generator 302 can be incorporated are shown in FIGS. 14-17 above,although these examples are not limiting.

By coupling the EM field generator 302 to the robotic arm 212, thekinematics of the robotic arm 212 can be used to provide a registrationbetween an EM coordinate frame 308 of the EM field generator 302 and therobotic coordinate frame 216 and/or global coordinate frame. That is,the kinematics of the robotic arm 212 (which as noted above are known)can be used to relate the EM coordinate frame 308 of the EM fieldgenerator 302 to the robotic coordinate frame 216 or global coordinateframe. This can provide one or more advantages over robotic medicalsystems that include separate EM field generators 202, such as therobotic system 200 of FIG. 22A.

One advantage can be that, by coupling the EM field generator 302 to therobotic arm 212, the need for a separate registration step (which asnoted above may require user input) can be eliminated. For example, withthe robotic medical system 300, it may not be necessary to manuallyperform a separate registration step, such as the registration stepsdescribed with reference to FIGS. 22B and 22C, in order to register theEM coordinate frame 308 to the robotic coordinate frame 216 or globalcoordinate frame. In the robotic medical system 300, such a registrationcan occur automatically (e.g., can be performed by the system), simplyby attaching the EM field generator 302 to the robotic arm 212. Therobotic medical system 300 can then use the known kinematics of therobotic arm 212 to relate to the EM coordinate frame 308 and the roboticcoordinate frame 216 or global coordinate frame.

Another advantage that can be achieved by coupling the EM fieldgenerator 302 to the robotic arm 212 is that the accuracy of theregistration between the EM coordinate frame 308 and the roboticcoordinate frame 216 can be improved. As noted above, in systems such asthe robotic system 200 of FIG. 22A, the registration step can introduceinaccuracies into the system, for example, due to inaccuracies innavigating during the manual registration step. In the robotic medicalsystem 300, the accuracy of the registration between the EM coordinateframe 308 and the robotic coordinate frame 216 is largely determined bythe accuracy of the kinematics of the robotic arms 212. When these areknown and well defined, the registration between the EM coordinate frame308 and the robotic coordinate frame 216 can be highly accurate.

A further advantage provided by some embodiments that include couplingthe EM field generator 302 to the robotic arm 212, may be that theposition of the EM field generator 302 can be adjusted using the roboticarm 212. This, in itself, can provide one or more advantages. For one,the robotic arm 212 can be used to move or reposition the EM fieldgenerator 302. This can, for example, allow the operator to adjust theposition of the working volume 306 of the magnetic field of the EM fieldgenerator 302. As will be described below, the robotic arm 212 can beused to move the EM field generator 302 such that it tracks the motionof, for example, the instrument 214 as it moves through the body.Additionally, the EM field generator 302 may be moved or repositioned byrobotic arm 212 to facilitate access to the patient, for example, toallow a fluoroscopic C-arm to access the patient. For example, therobotic arm 212 can be used to move the EM field generator 302 out ofthe way. Further, as the robotic arm 212 moves the EM field generator302, the relationship between the EM coordinate frame 308 and therobotic coordinate frame 216, which is determined by the kinematics ofthe robotic arm 212, remains known. Thus, it may not be necessary toreregister the EM coordinate frame 308 to the robotic coordinate frame216 after movement of the EM field generator 302.

Another advantage achievable by coupling the EM field generator 302 tothe robotic arm 212 may include increased accuracy of the determinedpositions of the EM sensor 204 because the orientation and distancebetween the EM field generator 302 and the EM sensors 204 can beadjusted to improve accuracy. For example, the accuracy of a determinedposition of an EM sensor 204 may decrease closer to the edges of theworking volume 306. If an EM sensor 204 is determined to be closer to anedge of the working volume 306, the robotic arm 212 can move the EMfield generator 302 such that the EM sensor 204 is positioned closer tothe center of the working volume 306 (or way from the edge of theworking volume 306), where the accuracy of the determined position maybe increased. As another example, the robotic arm 212 can move the EMfield generator 302 closer to an EM sensor 204 in order to focus in onit and track its position more closely (referred to herein as EMfocusing).

Additionally, because the robotic arm 212 can readily adjust theposition of the EM field generator 302 without requiring a separateregistration step, smaller EM field generators can be used, whencompared to other robotic systems (such as the robotic system 200) thatgenerally use stationary EM field generators. When stationary EM fieldgenerators are used, they must be large enough to provide a sufficientlylarge working volume to cover the medical site. Even so, they may notcover all the portions of the patient's anatomy, such as portions of thepatient's anatomy that are navigated through to access the medical site.By coupling the EM field generator 302 to the robotic arm 212, smallerEM field generators, which may in some cases have smaller workingvolumes can be used, because the position of the working volume 306 canbe adjusted by moving the EM field generator 302 with the robotic arm202. In some embodiments, this can allow the use of a compact fieldgenerator (cFG). Use of a compact field generator is not required,however, and larger field generators can be coupled to the robotic armin some embodiments.

When coupled to the robotic arm 212, the EM field generator 302 mayfunction in a similar manner to the EM field generator 202 previouslydescribed. For example, in the illustrated embodiment of the system 300of FIG. 23, one or more of the instruments 214 can include one or moreEM sensor(s) 204 positioned at a distal end thereof and/or on otherportions of the instruments 214. Additionally, the illustratedembodiment includes a second EM sensor 204 attached to the patient'schest, which can be used to track motion caused by the patient'srespiration. When the EM sensors 204 are positioned within the workingvolume 206 of the EM field generator 302, the positions of the EMsensors 204 within the EM coordinate frame 308 can be determined.

Additionally, as the EM coordinate frame 308 can be registered to therobotic coordinate frame 216 using the kinematics of the robotic arm212, the positions of the EM sensors 204 within the robotic coordinateframe 216 or the global coordinate frame can be determined. Byphysically connecting the EM field generator 302 to the robotic arm 212,the EM coordinate frame 308 and the robotic coordinate frame 216 orglobal coordinate frame can be linked together such that the positionsof the EM sensors 204 can be determined within the robotic coordinateframe 216 or global coordinate frame.

For example, in the illustrated embodiment of FIG. 23, the roboticmedical system 300 may include the EM field generator 302 configured togenerate an EM field. The robotic medical system 300 may include roboticarms 212. A first robotic arm 212 may be coupled to the EM fieldgenerator 302 and configured to articulate to move the EM fieldgenerator 302. The robotic medical system 300 may also include one moreprocessors configured to determine a position, in the EM coordinateframe 302 associated with the EM field generator 302, of an EM sensor204 within the EM field. The processor(s) may also be configured todetermine a registration between the EM coordinate frame 308 and therobotic coordinate frame 216 associated with the first robotic arm 212based on determining the position of the EM field generator 302 withinthe robotic coordinate frame. This can be accomplished, as describedabove, by using the kinematics of the robotic arm 212. Based on theregistration, the processor(s) can further be configured to determinethe position of the EM sensor 204 in the robotic coordinate frame 216.

As another example, in the illustrated embodiment of FIG. 23, therobotic medical system 300 can include a controller or control circuitcomprising one or more processors that are configured to controlmovement of the one or more robotic arms 212. The robotic arm 212 can beconfigured to articulate in response to receiving a command from thecontrol circuit. The EM field generator 302 can be coupled to therobotic arm 212, such that the EM field generator also moves in responseto one or more commands from the control circuit.

Besides the robotically controllable EM field generator 302 that iscoupled to the robotic arm 212, the illustrated embodiment of therobotic system 300 of FIG. 23 may be similar in many respects to therobotic system 200 described above with reference to FIG. 22A, whichincludes a separate EM field generator 202, or other robotic medicalsystems described throughout this application or elsewhere. For example,in the illustrated embodiment, the robotic medical system 300 comprisesthe cart 210 including two robotic arms 212. Although illustrated withtwo robotic arms 212, the robotic system 300 may use other numbers ofrobotic arms 212. For example, FIG. 2 (described above) illustrates anexample cart 11 that includes three robotic arms 12. Additionally, insome embodiments, the robotic arms 212 need not be attached to the cart210. In some embodiments, the robotic arms can be coupled to the bed orpatient platform 215, for example, as shown in FIGS. 5-10 and 14(described above). In some embodiments, the robotic medical system 300may comprise a plurality of robotic arms 212 (e.g., two, three, four,five six, or more robotic arms 212) that are coupled to the cart 210,the patient platform, and/or other components of the system.

Moreover, although the robotic system 300 is illustrated in FIG. 23 witha single tool or instrument 214 (e.g., a laparoscopic or endoscopicinstrument) coupled to one of the robotic arms 212, it should beappreciated that other numbers of instruments 214 (e.g., two, three,four, five, six or more instruments, including laparoscopic, endoscopic,and cameras) can be used in other embodiments.

FIG. 24 illustrates an embodiment of the EM field generator 302. In theillustrated embodiment, the EM field generator 302 is configured tocouple to an instrument drive mechanism 402 that can be positioned on arobotic arm, such that the EM field generator 302 is attachable to therobotic arm 212. As described above with reference to FIG. 16,instrument drive mechanisms, such as the instrument drive mechanism 402,can be configured with an interface 404 adapted to attach to varioustools or instruments for use by a robotic medical system. The EM fieldgenerator 302 can be configured with a corresponding interface 320 thatis configured to couple to the interface 404 of the instrument drivemechanism 402. In this way, the EM field generator 302 can couple to therobotic arms 212 in a similar manner as the other robotic tools orinstruments of the robotic system.

For example, the interface 320 of the EM field generator 302 an includeone or more connectors 322 configured to couple to one or morecorresponding connectors 406 on the interface 404 of the instrumentdrive mechanism 402. Such an arrangement can allow an operator easilyremovably couple the EM field generator 302 to the instrument drivemechanism 402.

In the illustrated embodiment, the EM field generator 302 includes ahousing 324. A proximal end 326 of the housing 234 can be configured tocouple to the instrument drive mechanism 402. For example, the proximalend 326 can include the interface 320 and connectors 322. The housing324 can extend from the proximal end 326 to a distal end 328. The distalend 328 can include a EM field generator unit 330 that is configured togenerate a magnetic field. In the illustrated embodiment, the distal end328 includes clips 332 configure to secure the EM field generator unit330. Other mechanisms or methods for securing the EM field generatorunit 330 are possible. In some embodiments, for example, as illustrated,the EM field generator unit 330 can be removable from the housing 324.In some embodiments, the EM field generator unit 330 is integrated intothe housing 324. In the illustrated embodiment, the EM field generatorunit 330 comprises a cFG, although other types of field generators mayalso be used.

The size and shape of the housing 326 can be determined or selected suchthat the kinematic relationship between the EM field generator unit 330and the robotic arm to which the EM field generator 302 is attached isknown, such that the kinematic registration between the EM coordinateframe associated with the EM field generator and the robotic coordinateframe associate with the robotic arm can be determined.

In the illustrated embodiment, the EM field generator unit 330 comprisesone or more connectors 334. The connectors 334 may electrically connectthe EM field generator unit 330 to a robotic system such that therobotic system can communicate with the EM field generator unit. Inother embodiments, electrical connections may be made between theinterfaces 320, 404 of the EM field generator 302 and instrument drivemechanism 402.

FIG. 25 illustrates how the EM field generator 302 can be attached tothe instrument drive mechanism 402 according to one embodiment. In theillustrated embodiment, the instrument drive mechanism 402 is showncovered with a sterile drape. A sterile adapter 410 is illustratedpositioned over the interface 404 of the instrument drive mechanism 402.The interface 320 of the EM field generator 302 can attach to thesterile adapter 410 such that the EM field generator 302 is coupled tothe instrument drive mechanism 402 with the sterile adapter 410positioned therebetween. As shown in FIG. 25, the EM field generatorunit 330 may also be covered with a sterile drape.

FIG. 26 is a flow chart depicting an embodiment of a method 500 forperforming a robotic medical procedure using a EM field generatorconfigured to couple to (or otherwise connect or integrate into) arobotic arm. In the illustrated embodiment, the method begins at block502. Block 502 can include generating an EM field with an EM fieldgenerator coupled to a first robotic arm. The EM field can be associatedwith an EM coordinate frame. Next, the method 500 moves to block 504, atwhich a registration between the EM coordinate frame and a roboticcoordinate frame associated with the first robotic arm is determined.The registration can be based on determining a position of the EM fieldgenerator within the robotic coordinate frame based on kinematics of thefirst robotic arm. At block 506, a position of an EM sensor within theEM coordinate frame is determined. At block 508, the method includesdetermining the position of an EM sensor in the robotic coordinate framebased on the registration determined at block 504.

In some embodiments, the method 500 may optionally include removablycoupling the EM field generator to the first robotic arm, for example,as shown in FIG. 25. The method 500 may also optionally include movingthe EM field generator with the first robotic arm. Moving the EM fieldgenerator may be useful in that the working volume of the EM field canbe positioned as needed to facilitate the procedure and/or the EM fieldgenerator can be moved out of the way to allow access to the patientduring a medical procedure. Moving the EM field generator with the firstrobotic arm to adjust a position of the EM field generator relative tothe EM sensor may also improve the accuracy of the determined positionof the EM sensor within EM field coordinate frame (and correspondinglyin the robotic coordinate frame through the registration).

The method 500 may also include moving a medical instrument coupled to asecond robotic arm, wherein the robotic coordinate frame is alsoassociated with the second robotic arm. The EM sensor can be positionedon the medical instrument, such that the guidance of navigation of thesecond instrument can be determined based, at least in part, on thedetermined position of the EM sensor.

C. Example Applications for Robotically Controllable Field Generators

In addition to providing one or more of the benefits discussed above,robotically controllable field generators coupled or otherwiseintegrated into robotic arms can be used to enable, facilitate, and/orimprove various functionality during a robotic medical procedure. Thissection outlines several example applications for roboticallycontrollable field generators that can be advantageous over otherrobotic systems that include, for example, static or stationary fieldgenerator.

i. Field Generator Setup and Instrument Tracking

Robotically controllable field generators coupled to or integrated intorobotic arms as described above can be used to facilitate fieldgenerator setup and instrument tracking. Several examples are providedin this section to illustrate, without limiting, these concepts.

As used herein, “setup” of the EM field generator refers to determiningwhere to position the EM field generator to facilitate a procedure. Theposition may be based on, for example, the determined positions of oneor more EM sensors. In some instances, the position may be adjustedduring the procedure, for example as one or more of the EM sensors aremoved during the procedure. As noted above with reference to FIGS. 21and 22A, the EM field generator 202 would be positioned relative to apatient such that the working volume 206 of the field generator 202overlaps with a portion of the patient's anatomy at which the medicalprocedure is performed (e.g., a medical site). The EM field generator202 could be supported by a stand or other support structure such as thepatient platform 215. Once positioned, registration steps (such as thosedescribed with reference to FIGS. 22B and 22C) would be performed toregister the EM coordinate frame 208 associated with the EM fieldgenerator 202 to the robotic coordinate frame 216. Further, during theprocedure, the EM field generator 202 generally was not moved becausemovement would need to be done manually and would require that theregistration steps be repeated to reregister the EM coordinate frame 208associated with the EM field generator 202 to the robotic coordinateframe 216. Robotically-controllable EM field generators that can berepositioned using robotic arms of the system facilitate the setupprocedure by allowing the system to determine locations at which toposition the EM field generator and moving the EM field generator tothose locations using the robotic arms of the system. Further, duringthe procedure, as the positions of tools and instruments that include EMsensors change, the system can adjust the position of the EM fieldgenerator such that it remains positioned in advantageous locations.

FIGS. 28A and 28B illustrate an introductory example that shows that therobotically controlled EM field generator 302 (described, for example,above with reference to FIGS. 23-25) can be used to facilitate set upand positioning of various tools and devices used for a medicalprocedure. Additional examples will be described in more detail furtherbelow. FIG. 28A illustrates three EM position sensors 204, as well asthe EM field generator 302 coupled to a robotic arm 212. The EM positionsensors 204 can be, for example, positioned on medical instruments (suchas endoscopic, laparoscopic, and/or other types of surgical tools) ordirectly on the patient (such as, for example, EM patch sensorsconfigured to track patient movement or respiration). For ease ofillustration, the medical instruments are not illustrated.

As shown in FIG. 28A, the system can be configured to move the EM fieldgenerator 302 using the robotic arm 212 to identify the locations of theEM sensors 204. Such movement can include translation, rotation, and/oroscillation of the EM field generator 302. In FIG. 28A, an examplecircular path is illustrated. Once the positions of the EM sensors 202are determined, the robotic arm 212 can move to reposition the EM fieldgenerator 302 at a comparatively improved (referred herein as a “workinglocation”) or centralized location at which the EM sensors 204 arepositioned within the working volume 306 of the EM field generator 302,as shown, for example, in FIG. 28B. This can facilitate setup as thesystem can determine an working or central location at which to positionthe EM field generator 302 automatically. In related aspects, theposition of one or more EM sensors 204 may be adjusted to achieve acomparatively improved arrangement of the EM sensors 204 based on therange of movement or oscillation achievable by the EM field generator,the medical procedural steps, and/or the anatomical characteristics ofthe patient.

In some embodiments, the system locates and determines the positionand/or orientation of the EM sensors 204 and then computes the centroidor the geometric center of those positions or a shape defined by one ormore of the positions. The EM field generator 302 can then bepositioned, using the robotic arm 212, such that the center of theworking volume 306 is aligned with, for example, the determined centroidof the positions of the EM sensors 204.

In addition, FIG. 28A illustrates that the EM field generator 302 canincrease the functional size of the working volume 306. For example, ifEM sensors 204 are separated such that they cannot all be fit within theworking volume 306, the system can move the EM field generator (in ascanning motion) to produce a functional working volume 306 that islarge enough to track all of the EM sensors 204. In some embodiments,the system may prioritize certain EM sensors 204 such that they remainconstantly positioned within the working volume 306, while other EMsensors 204 are sometimes inside and sometimes outside of the workingvolume 306.

“Instrument tracking,” as used herein, generally refers to moving therobotically controllable EM field generator using a robotic arm to trackor follow motion of an EM sensor. As noted above, prior systemsgenerally involved the use of stationary EM field generators that couldnot or (would not, in order to avoid re-registration) be moved to trackor follow motion of an EM sensor. In such systems, if an EM sensor wasmoved outside of the working volume of the EM field generator, the EMsensor would no longer be detectable until it was moved back into theworking volume. Robotically controllable field generators that can bemoved with robotic arms can facilitate instrument tracking by allowingthe EM field generator to be moved or otherwise adjusted to track motionof one or more instrument sensors. Thus, rather than having an EM sensormove outside of the working volume of the EM field generator, the EMfield generator can be moved along with the EM sensor such that the EMsensor remains positioned within the working volume, facilitatingcontinual tracking of the EM sensor.

FIGS. 27A and 27B illustrate an introductory embodiment of automaticinstrument tracking that can be facilitated using a roboticallycontrollable field generator, such as the EM field generator 302described above with reference to FIGS. 23-25. In particular, FIGS. 27Aand 27B illustrate an example of automatic instrument tracking during anexample medical procedure, such as a ureteroscopic procedure, duringwhich a ureteroscope 502 is robotically navigated through a patientorifice, through the ureter, and into a kidney. Although a ureteroscopicprocedure is illustrated, automatic instrument tracking can be used inother types of procedures, including endoscopic and/or laparoscopicprocedures. FIG. 27A illustrates the procedure at an earlier or firsttime/step (e.g., just after insertion of the ureteroscope 502) and FIG.27B illustrates the procedure at a later or second time/step (e.g.,after the distal tip of the ureteroscope 502 has been navigated into thekidney).

As illustrated in FIGS. 27A and 27B, the ureteroscope 502 can be coupledto one or more robotic arms 212 that are configured to manipulate theureteroscope 502 and perform the insertion thereof as described, forexample, with respect to FIG. 3. In FIGS. 27A and 27B, the EM fieldgenerator 302 is coupled to a third robotic arm 212 that is configuredto move to adjust the position of the EM field generator 302 (andcorrespondingly, the position of the working volume 306 of the magneticfield generated by the EM field generator 302). As described above,because the EM field generator 302 is attached to the robotic arm 212,the EM coordinate frame associated with the EM field generator 302 canbe automatically registered to the robotic coordinate frame associatedwith the robotic arms 212 through the kinematics of the robotic arm 212to which the EM field generator 302 is attached.

As shown in FIGS. 27A and 27B, the system can be configured to move theEM field generator 302 with the corresponding robotic arm 212 such thatit automatically tracks the position of the ureteroscope 502. In theillustrated embodiment, the ureteroscope 502 includes an EM sensor 204positioned on the distal tip thereof. As described above, the positionof the EM sensor 204 can be determined relative to the EM coordinateframe when the EM sensor 204 is positioned within the working volume 306of the EM field generator 302. Further, because the EM coordinate framecan be registered to the robotic coordinate frame, the position of theEM sensor 204 within the robotic coordinate frame is also determinable.

As the ureteroscope 502 is inserted further into the patient, therobotic arm 212 to which the EM field generator is attached can move oradjust to automatically reposition the EM field generator such that theposition sensor 204 remains positioned within the working volume 306 ofthe EM field generator. In some embodiments, this automatic tracking ofthe ureteroscope 502 is configured such that the EM field generator 302is moved or adjusted so that the EM sensor 204 remains positioned at thecenter of the working volume 306, although this need not be the case inall embodiments. Additionally, in some embodiments, the orientation ofthe EM field generator 302 can be adjusted relative to the EM sensor 204to provide for optimal or improved accuracy in determining the positionthereof.

This type of automatic tracking can advantageously allow that theposition of the ureteroscope 502 be tracked throughout the procedure(e.g., from insertion to work within the kidney). This may not bepossible in robotic systems that use a static or stationary fieldgenerator. As noted above, static or stationary field generators areoften set up such that their working volume is centered on the primarymedical site (e.g., the kidney). In this position, the working volumemay not extend sufficiently to provide coverage of insertion andnavigation through the ureter. Thus, in systems that use static orstationary field generators, operators may not be able to utilize EMbased navigation until they have navigated the instrument into theworking volume of the field generator. Moving a static or stationaryfield generator to follow an instrument during insertion and navigationof an instrument to the medical site is also generally not possible as aregistration step would need to be re-performed each time the fieldgenerator is moved, which is impractical and may cause delays during themedical procedure, potentially negatively impacting the outcome of theprocedure.

As shown in FIGS. 27A and 27B, however, the robotically controlled EMfield generator 302 can readily track the position of the ureteroscope502 from insertion to working within the kidney as the robotic arm 212can reposition the EM field generator 302 as needed, and the kinematicsof the robotic arm 212 can be used to provide a continuous registrationbetween the EM coordinate frame and the robotic coordinate frame.

An additional advantage that can be achieved using the automaticinstrument tracking shown in FIGS. 27A and 27B is improved anatomicalmapping. As an instrument is navigated through a patient's anatomy, thepositions of an EM sensor on the instrument can be used to build ananatomical map. For example, in the case of bronchoscopy, as abronchoscope including an EM sensor is navigated through the patient'sairways, the determined positions of the EM sensor can be used toproduce a map of the patient's lungs. With static or stationary EM fieldgenerators, this is only possible for regions within the working volumeof the EM field generator. However, through the use of a roboticallycontrollable EM field generator 302 that allows for a robotic arm 212 toreposition the EM field generator 302, the capability of generatinganatomical maps is increased.

Using the example of FIGS. 27A and 27B, the position of the EM sensor204 can be determined from insertion (FIG. 27A) to within the kidney(FIG. 27B) producing a generally complete anatomical map of thepatient's urinary tract. In contrast, if a static field generator wereused with its working volume centered on the kidney, EM sensor datamight not be available that could be used to map the patient's ureter.Accordingly, use of the robotically controllable EM field generator 302as described herein can extend the ability to produce anatomical mappingusing EM sensor data, leading to more complete anatomical maps.

FIG. 29 is a block diagram representative of a robotic medical system300, such as the example illustrated in FIG. 22A, which includes arobotically controllable EM field generator 302 mounted on a robotic arm212. As shown in FIG. 29 (and also in FIG. 23), the system 300 caninclude one or more robotic arms 212. In the illustrated embodiment, thesystem 300 includes a first robotic arm 212 and a second robotic arm212, although in other embodiments, other numbers of robotic arms (e.g.,one, two, three, four, five, six or more) can be included. As shown, thefirst robotic arm 212 is coupled to an EM field generator 302. Asdescribed above, the EM field generator 302 can be attached or coupledto the first robotic arm 212 or it can be integrated directly into thefirst robotic arm 212 itself. The EM field generator 302 is configuredto generate an EM field having a working volume 306, within which thepositions of EM sensors 204 can be determined relative to an EMcoordinate frame 308. The first robotic arm 212 can be configured to beable to adjust the position of the EM field generator 302. For example,the first robotic arm 212 can move (e.g., change pose or shape) toreadjust the positon of the EM field generator 302, and correspondingly,the position of the working volume 306 of the EM field.

The first robotic arm 212 (as well as the other robotic arms 212 of thesystem) is associated with a robotic coordinate frame 216 as describedabove. Based on the kinematic position of the arm 212, the position ofthe EM field generator 302 within the robotic coordinate frame 216 canbe determined. This allows for a registration or mapping between the EMcoordinate frame 308 and the robotic coordinate frame 216 to bedetermined as described above. As shown in FIG. 29 (and also in FIG.23), the system 300 can include one or more EM sensors 204. Thepositions of the EM sensors 204 can be determined relative to the EMcoordinate frame 308 and mapped to the robotic coordinate frame 216using the registration previously described.

In FIG. 29 (and FIG. 23), one of the EM sensors 204 is positioned on amedical instrument 214. The medical instrument 214 can be, in someembodiments, coupled to the second robotic arm 212. This can allow thesecond robotic arm 212 to manipulate and control the medical instrument214. As the second robotic arm 212 manipulates and controls the medicalinstrument 214, the position of the medical instrument 214 can bedetermined based on the determined position of the EM sensor 204. Asshown in FIG. 29, the system 300 can include additional EM sensors 204that can be tracked as well.

The block diagram of FIG. 29 further illustrates that the system 300 caninclude a processor 380 and a memory 382. The memory 382 can be used tostore instructions that when executed by the processor 380 can enablevarious functionality of the system 300, such as determining and usingthe EM coordinate frame 308 to robotic coordinate frame 216 registrationand mapping, setup of the EM field generator 302, and tracking of themedical instrument 214 as described in this section and throughout thisapplication. For example, the processor 380 can be in communication withthe first robotic arm 212 and the EM field generator 302 and configuredto determine a position of one or more of the EM sensors 204 within theEM field generated by the EM field generator 302, and adjust theposition of the EM field generator 302 by commanding movement of thefirst robotic arm 212 based on the determined position of the EM sensor204. The commanded movement of the first robotic arm 212 based on thedetermined position of the EM sensor 204 can be used to setup theposition of the EM field generator 302 or to track the position of theEM sensor 204 as it moves during the procedure.

In some embodiments, the processor 302 is configured to first determinethe position of the EM sensor 204 relative to the EM coordinate frame308, and then map that position into the robotic coordinate frame 216.As described above, this can be accomplished by, for example, (i)determining the position of the EM sensor 204 within the EM fieldrelative to the EM coordinate frame 308 associated with EM fieldgenerator 302, (ii) determining a registration between the EM coordinateframe 308 and the robotic coordinate frame 216 associated with the firstrobotic arm 212 based on determining a position of the EM fieldgenerator within the robotic coordinate frame 216, and (iii) based onthe registration, determining the position of the EM sensor 204 in therobotic coordinate frame 216. Determining of the position of the EMfield generator 302 within the robotic coordinate frame 216 at step (ii)can be based on kinematics of the first robotic arm 212.

FIGS. 30A-33D illustrate various examples or approaches to moving the EMfield generator 302 with the first robotic arm 204 based on thedetermined position of the EM sensor 204 in order to facilitate fieldgenerator setup and instrument tracking.

FIGS. 30A and 30B are perspective views that illustrate an example ofmoving an EM field generator with a robotic arm such that an EM sensoris positioned at a predetermined position within an EM field. This canoccur, for example, during field generator setup and can advantageouslybe used to position the EM field generator 302 relative to an EM sensor204. In some embodiments, it can be advantageous to setup or align theEM field generator 302 such that the EM sensor 204 is located at apredetermined position with the working volume 306 of the EM field. Thiscan be useful for several reasons. For example, in some instances, thiscan center the working volume 306 of the EM field around the EM sensor204. This can help ensure that, if the EM sensor 204 is moved, it canstill be tracked as it can move without moving outside of the workingvolume 306. As another example, this can increase the accuracy of thedetermined position of the EM sensor 204 because, in some instances, theability to determine the position of the EM sensor 204 within the EMfield may be more accurate toward the center of the working volume 306than on the extremities of the working volume 306. Accordingly, in someembodiments, the predetermined position comprises the center of theworking volume 306, and the EM field generator 302 is moved such thatthe EM sensor 204 is positioned at the center of the working volume 306as shown in FIGS. 30A and 30B.

In FIGS. 30A and 30B, the predetermined position 384 is the center ofthe working volume 306 and is represented in the figures as an x. FIG.30A illustrates the system in a first state, prior to moving the EMfield generator 302 such that the EM sensor 204 is positioned at thepredetermined position 384. As shown in FIG. 30A, the EM sensor 204 ispositioned within the working volume 306 of the EM field generator 302,but it is not located at the predetermined position 384. To position theEM sensor 204 at the predetermined position 384, the EM field generator302 must be moved in the direction of the arrows shown, causing acorresponding movement of the working volume 306 and the predeterminedposition 384. FIG. 30B illustrates the system in a second state, afterthe adjusting the position of the EM field generator 302 based on thedetermined position of the EM sensor 204 to causes the EM sensor 204 tobe positioned at a predetermined position 384 within the EM field. Inthis example, the EM field generator 302 can now be considered “setup”as its position has been adjusted relative to the EM sensor 204.Notably, the movement of the EM field generator 302 can be accomplishedwith the robotic arm 212 to which the EM field generator 302 isattached. Further, points within the EM coordinate frame, such as theposition of the EM sensor 204 and the predetermined position 384 can bemapped into the robotic coordinate frame using the registration based onthe kinematics of the robotic arm such that all positions (e.g., EMsensor positions, the predetermined positon 384, and the position of theEM field generator 302) can be processed in a single space (e.g., therobotic coordinate frame).

In FIGS. 30A and 30B the predetermined position within 384 within theworking volume 306 of the EM field comprises the center of a workingvolume 306. This, however, need not be the case in all embodiments, andother predetermined positions 384 within the working volume 306 can beused. For example, another possible predetermined position 384 maycomprise a position located on one extreme of the working volume 306such that the EM sensor 204 can be moved entirely across the workingvolume 306 before the EM field generator 302 would need to berepositioned to continue tracking the EM sensor 204.

In some embodiments, the EM field generator 302 can be positioned suchthat the EM sensor 204 is positioned within a predetermined region orportion of the working volume 306 of the EM field, rather than at aspecific predetermined position. FIGS. 31A and 31B are perspective viewsthat illustrate an example of moving an EM field generator 302 with arobotic arm 212 such that the EM sensor 204 is positioned at within apredetermined region 386 within the working volume 306. In FIGS. 31A and31B, a predetermined region 386, representing a subsection of theworking volume 386 is illustrated. In the figures, both the workingvolume 306 and the predetermined region 386 are represented asrectangular prisms. This, however, is merely one example, and othershapes for the working volume and the predetermined region 386 arepossible. Further, the shape of the predetermined region 386 need notcorrespond to the shape of the working volume 306. For example, theworking volume 306 may comprise the shape of a rectangular prism and thepredetermined region 386 may comprise the shape of a sphere within theworking volume 306. In some embodiment, the predetermined region 386 mayrepresent a portion of the working volume 306 within which the positionsof EM sensors 204 can be determined with increased accuracy, and assuch, it may be desirable to positon the EM field generator 302 relativeto the EM sensor 204 such that the EM sensor 204 is within thepredetermined region 386. In some embodiments, the predetermined region386 may comprise the entirety of the working volume 306.

FIG. 31A illustrates the system in a first state, prior to adjusting theposition of the EM field generator 302 such that the EM sensor 204 ispositioned within the predetermined region 386. As shown, the EM sensor204 is positioned within the working volume 306 of the EM field, but itis not positioned within the predetermined region 386. In order toposition the working volume 306 such that the EM sensor 204 ispositioned within the predetermined region 386, the EM field generator302 can be moved, using the robotic arm 212 to which it is attached, inthe direction of the illustrated arrows. This motion causes acorresponding movement of the working volume 306 and the predeterminedregion 386. FIG. 31B illustrates the system in a second state, afteradjusting the position of the EM field generator 302 such that the EMsensor 204 is positioned within the predetermined region 386. As shown,the EM field generator 302 has been moved using the robotic arm 212 suchthe EM sensor 204 is now positioned within the predetermined region 386.As before, the registration described above can map points in the EMcoordinate frame to corresponding points in the robotic coordinate framebased on the kinematics of the arm such that all positions can berepresented in a single space.

The examples of FIGS. 31A-31B and 32A-32B involve adjusting the positionof the EM field generator 302 relative to the EM sensor 204. In someembodiments, the system may, additionally or alternatively, determine anorientation of the EM sensor 204 within the EM field, and adjust atleast one of an orientation and the position of the EM field generator302 by commanding movement of the first robotic arm 212 based on thedetermined orientation of the EM sensor 204. In certain instances, theaccuracy with which the position of the EM sensor 204 can be determinedwithin the working volume 306 may be influenced or affected by theorientation of the EM sensor 204 relative to the EM field generator 302.Accordingly, in some embodiments, the processor 380 is configured toadjust the at least one of the orientation and position of the EM fieldgenerator based on the determined orientation of the EM sensor toincrease an accuracy of the determined position of the EM sensor 204within the EM field. This can involve, for example, adjusting the pitch,yaw, and/or roll of the EM field generator 302 relative to the EM sensor204 using the robotic arm 212 to which the EM field generator 302 isattached.

As described above, the examples of FIGS. 31A-31B and 32A-32B havegenerally related to determining a position for the EM field generator302 relative to a determined position of an EM sensor 204 in order tosetup the EM field generator 302 for a procedure. This setup isfacilitated by mounting the EM field generator 302 on the robotic arm212 such that its position can be robotically adjusted. Roboticadjustment of the position of the EM field generator 302 alsofacilitates instrument tracking, which can occur as one or more of theEM sensors 204 move during the procedure. As noted previously, one ormore of the EM sensors 204 may be positioned on medical tools orinstruments that are moved during the procedure. The EM sensors 204 canallow tracking of the movement of these tools. As described in thefollowing examples, as the movement of the EM sensors 204 is detected,new positions for the EM field generator 302 can be determined, and theEM field generator 302 can be moved using the robotic arm 212 to whichit is attached.

FIGS. 32A and 32B are perspective views that illustrate an example ofmoving an EM field generator 302 with a robotic arm 212 along a path 390that tracks a path 388 of movement of an EM sensor 204. In this manner,the path 390 of the field generator 302 tracks, follows, or mirrors thepath 388 of the EM sensor 204. In the example of FIGS. 32A and 32B, thesystem is configured to maintain the position of the EM field generator302 such that the EM sensor 204 remains positioned at the predeterminedlocation 384 (in the illustrated example, the center of the workingvolume 306) as the EM sensor 204 is moved along the path 388.

During a medical procedure, a medical instrument (such as the medicalinstrument 214 of FIGS. 23 and 29) that includes the EM sensor 204positioned thereon may be navigated through the patient's anatomy. Thismotion of the EM sensor 204 (and the corresponding instrument 214) isrepresented by the path 388. FIG. 32A illustrates the system with the EMsensor 204 positioned at a first location along the path 388. If the EMsensor 204 were to continue along the path 388 while the EM fieldgenerator 302 remains stationary, the EM sensor 204 would no longer bepositioned at the predetermined position 384 within the working volume306. To maintain the EM sensor 204 at the predetermined position 384within the working volume 306, the EM field generator 302 must be movedalong a corresponding path 390. In this manner, the EM field generator302 tracks or follows the movement of the EM sensor 204. FIG. 32Billustrates the system after movement of the EM sensor 204 along itspath 388. Because the EM field generator 302 has also moved along thecorresponding path 390, the EM sensor 204 remains positioned at thepredetermined position 384 within the working volume 306. This isenabled by moving the EM field generator 302 with the robotic arm 212 towhich it is attached.

In some embodiments, tracking of the EM sensor 204 with the EM fieldgenerator 302 need not have a direct correspondence. For example, thepath 390 of the EM field generator 302 need not directly correspond tothe path 388 of the EM sensor 204. FIGS. 33A-33D illustrate an example.

FIGS. 33A-33D are perspective views that illustrate an example of movingthe EM field generator 302 with a robotic arm 212 such that the EMsensor 204, which is moving along the path 388 remains positioned withinthe predetermined region 386 of the working volume 306. As will beshown, in this example, the EM field generator 302 tracks the movementof the EM sensor 204 but is not moved along a path that directlycorresponds to the path 388 of the EM sensor 204. In this example, thesystem is configured to adjust the position of the EM field generator302 when the EM sensor 204 is moved to the boundary or outside of thepredetermined region 386 of the working volume 306. For example, theprocessor 380 can be configured to command movement of the EM fieldgenerator 302 with the first robotic arm 212 during movement of themedical instrument 214 that includes EM sensor 204 such that the EMsensor 204 remains positioned within the predetermined region 386 of theworking volume 306 of the EM field.

FIG. 33A illustrates the system in a first state. As shown, the EMsensor 204 is positioned within the predetermined region 386 of theworking volume 306 and traveling along the path 388. FIG. 33Billustrates the system at a second state. As shown, the EM sensor 204has continued along the path 204 but is still positioned within thepredetermined region 386. Since the EM sensor 204 is still positionedwithin the predetermined region 386, the EM field generator 302 has notyet been moved. FIG. 33C illustrates a third state, wherein the EMsensor 204 has reached an edge or boundary of the predetermined region386. If the EM sensor 204 continues along the path 388 while the EMfield generator 302 remains stationary, the EM sensor 204 will moveoutside of the predetermined region 386. To maintain the EM sensor 204within the predetermined region 386, the EM field generator 302 must bemoved in the direction of the illustrated arrow. This movement can bedetermined, for example, by the processor 380, which can command therobotic arm 212 to move the EM field generator 302. FIG. 33D illustratesthe system in a fourth state, after the EM field generator 302 has beenmoved. As shown, the movement can reposition the EM field generator 302such that the EM sensor 204 is once again positioned within thepredetermined region 386, even as the EM sensor 204 continues along thepath 388. In this way, the EM field generator 302 is once again moved totrack or follow the movement of the EM sensor 204, even though themovement of the EM field generator 302 does not directly correspond tothe movement of the EM sensor 204 as in the example of FIGS. 32A and32B.

FIG. 34 is a flowchart providing an example method 400 for moving an EMfield generator coupled to a robotic arm based on a determined positionof an EM sensor within an EM field. The method 400 can, for example, beexecuted by the processor 380 to provide field generator setup andinstrument tracking. The method 400 begins at block 401, which involvesdetermining a position of an EM sensor of a medical instrument within aworking volume of an EM field generated by an EM field generator coupledto a first robotic arm. The position of an EM sensor can be determinedrelative to an EM coordinate frame associated with the EM fieldgenerator. In some embodiments, the position of the EM sensor canfurther be determined relative to a robotic coordinate frame associatedwith the first robotic arm by applying a registration determined basedon the kinematics of the robotic arm as described above.

The method 400 include block 403, where medical instrument is moved.Movement of the medical instrument causes a corresponding movement ofthe EM sensor positioned thereon. In some embodiments, the medicalinstrument comprises a manually controlled instrument, and movement ofthe medical instrument is achieved manually. In other embodiments, themedical instruments comprises a robotically controllable instrument. Forexample, the medical instrument can be coupled to a second robotic arm,and moving the medical instrument can comprise moving the instrumentwith the second robotic arm. In some embodiments, moving the instrumentwith the second robotic arm comprises articulating the second roboticarm. The medical instrument can be coupled to an instrument drivemechanism, and moving the medical instrument with the second robotic armcan comprise actuating the medical instrument with the instrument drivemechanism. Movement of the instrument can be detected by determiningthat the position of an EM sensor attached to the instrument has moved.

The method 400 includes block 405, which involves, in response to themoving of the medical instrument, commanding the first robotic arm tomove the EM field generator such that the EM sensor remains positionedwithin the working volume of the EM field generator. For example, ifmovement of the medical instrument would move the EM sensor outside ofthe working volume, the EM field generator can be repositioned using thefirst robotic arm to maintain the EM sensor within the working volume ofthe EM field.

In some embodiments, commanding the first robotic arm to move the EMfield generator causes the EM field generator to track the movement ofthe medical instrument. For example, commanding the first robotic arm tomove the EM field generator can comprise moving the EM field generatorsuch that the EM sensor is positioned or remains positioned within apredetermined region of the working volume of the EM field as shown inthe examples of FIGS. 31A-31B and 33A-33D. As another example,commanding the first robotic arm to move the EM field generator cancomprise moving the EM field generator such that the EM sensor is orremains positioned at a predetermined position within the working volumeof the EM field as shown in the examples of FIGS. 30A-30B and 32A-32B.

In some embodiments, the method 400 can optionally include determiningan orientation of the EM sensor within the EM field, and adjusting atleast one of an orientation and the position of the EM field generatorby commanding movement of the first robotic arm based on the determinedorientation of the EM sensor. As described above, adjusting the at leastone of the orientation and position of the EM field may increaseaccuracy of the determined position of the EM sensor within the workingvolume of the EM field.

Additionally, the method 400 can be performed as a loop as indicated bydashed line 407 to continually track an EM sensor and adjust theposition of the EM field generator accordingly. That is, at a pluralityof discrete time steps, the position of the EM sensor can be determined,and the EM field generator can be readjusted based on each newlydetermined position.

The examples provided above have shown that a robotic system, such asthe system 300 shown in FIGS. 23 and 29, can be configured to move an EMfield generator 302 using a robotic arm 312 to setup the EM fieldgenerator 302 relative to one EM sensor 204 and/or to track movement ofone EM sensor 204. However, the system 300 is not limited to embodimentswith one EM sensor 204. For example, the processor 380 can be configuredto determine positions of a plurality of EM sensors 204 within the EMfield, determine a generator position for the EM field generator 302based on the determined positions of the plurality of EM sensors 204,and command the first robotic arm 212 to move the EM field generator 302to the generator position. That is, the system 300 can be configured forEM field generator setup and tracking based on a plurality of EM sensors204. Examples using based on a plurality of EM sensors 204 are providedwith respect to FIGS. 35A-37.

FIGS. 35A and 35B are perspective views that illustrate an example ofmoving an EM field generator 302 to a field generator position using arobotic arm 212 based on determined positions of a plurality of EMsensors 204 within an EM field. In some ways similar to earlierexamples, this can occur, during field generator setup and canadvantageously be used to position the EM field generator 302 relativeto the plurality of EM sensors 204.

FIG. 35A illustrates the system in a first state. As shown, in the firststate, the plurality of EM sensors 204 are positioned within the workingvolume 306 of the EM field generator 302. However, as illustrated, theplurality of EM sensors 204 are not centered within the working volume306. To center the plurality of EM sensors 204 within the working volume306, a field generator position 392 can be determined. The fieldgenerator position 392 can represent a position to which the EM fieldgenerator 302 can be moved to center the plurality of EM sensors 204within the working volume. In the illustrated embodiment, the generatorposition 392 is represented as an x. The EM field generator 302 must bemoved in the direction of the arrow shown, causing a correspondingmovement of the working volume 306, to arrive at the generator position392.

The generator position 392 can be determined based on the determinedpositions of the plurality of EM sensors 302. In one example, theprocessor 380 is configured to determine the generator position 392based on determining a centroid of the determined positions for theplurality of EM sensors 204. The generator position 392 can then bedetermined by finding the position to which the EM field generator 302should be moved such that the centroid of the positions of the pluralityof EM sensors 204 is positioned at a predetermined position (such as thecenter or other desired positions) within the working volume 306. Asbefore, the determined positions of the plurality of EM sensors can bemapped into the robotic coordinate frame using the registrationpreviously described based on the kinematics of the robotic arm 212 towhich the EM field generator 302 is attached.

FIG. 35B illustrates the system in a second state, after adjusting theposition of the EM field generator 302 to the generator position 392. Asshown, the plurality of EM sensors 204 are now positioned closer to thecenter of the working volume 206. Accordingly, the system can facilitatesetup and positioning of the EM field generator 302 relative to aplurality of EM sensors 204. Although this example has described“centering” the plurality of EM sensors 204 within the working volume306, the generator position 392 can be determined to position theplurality of EM sensors at other locations within the working volume306.

FIGS. 36A, 36B, and 36C are perspective views that illustrate an exampleof readjusting a field generator position 392 of the EM field generator302 using a robotic arm 212 based on the determined positions of aplurality of EM sensors 204 within the working volume 306, wherein atleast one of the plurality of EM sensors 204 is moving. For example, theprocessor 380 can be configured detect movement of at least one of theEM sensors 204, determine a new generator position 392 for the EM fieldgenerator 302 based on the detected movement, and command the firstrobotic arm 212 to move the EM field generator 302 to the new generatorposition 392.

As shown in FIG. 36A, one of the EM sensors 204 is moving along a path388. It may be desirable to adjust the position of the field generator302 based on this movement. For example, if not adjusted, the EM sensor204 may move along the path to a location that falls outside of theworking volume 306, at which point tracking of the EM sensor 204 wouldbe lost.

Accordingly, as shown in FIG. 36B, a new field generator position 392can be determined based on the positions of the plurality of EM sensors204 and considering the movement of the one EM sensor 204 along the path388. The new field generator position 392 can be, for example, based ona current centroid of the positions of the plurality of EM sensors 204.As shown, the EM field generator 302 would need to move in the directionof the illustrated arrow to reach the new generator position 392. FIG.36C illustrates the system after movement to the new field generatorposition 392. As shown, the positions of the plurality of EM sensors 204are now re-centered within the working volume 306. This canadvantageously allow the position of the EM field generator 302 to becontinually adjusted to optimize the position of the working volume 306such that the positions of the plurality of EM sensors 204 remainpositioned within the working volume 306 for as long as possible, evenas they move away from each other.

FIG. 37 is a flowchart illustrating an example method 410 fordetermining a field generator position for an EM field generator mountedon a robotic arm based on determined positions of a plurality of EMsensors within an EM field. The method 410 begins at block 411, whichinvolves generating an EM field with an EM field generator coupled to afirst robotic arm.

At block 413 of the method 410, embodiments may determine positions of aplurality of EM sensors within the EM field. This can be accomplished bydetermining the positions of the plurality of EM sensors within the EMfield relative to an EM coordinate frame associated with EM fieldgenerator, determining a registration between the EM coordinate frameand a robotic coordinate frame associated with the first robotic armbased on determining a position of the EM field generator within therobotic coordinate frame, and, based on the registration, determiningthe positions of the plurality of EM sensors in the robotic coordinateframe. The determination of the position of the EM field generatorwithin the robotic coordinate frame can be based on kinematics of thefirst robotic arm.

At block 415, the method 410 involves determining a generator positionfor the EM field generator based on the determined positions of theplurality of EM sensors. Determining the generator position can be basedon determining centroid of the positions of the plurality of EM sensors204. The EM field generator position can be a position that the EM fieldgenerator can be moved to that will desirably position the plurality ofEM sensors within the working volume of the EM field.

Block 417 includes commanding the first robotic arm to move the EM fieldgenerator to the generator position. For example, the system can movethe EM field generator to the generator position using the first roboticarm. As the EM field generator is moved, the position of the workingvolume is correspondingly repositioned relative to the positions of theEM sensors.

As illustrated by dashed line 419, the method 410 can be performed as aloop to continually adjust the position of the EM field generator asdescribed above with reference to FIGS. 36A-36C. For example, the method410 can also include detecting movement of at least one of the pluralityof the EM sensors, determining a new generator position for the EM fieldgenerator based on the detected movement, and commanding the firstrobotic arm to move the EM field generator to the new generatorposition.

ii. Field Generator Setup and Instrument Tracking with an ExpandedWorking Volume

In the previous section, robotic medical systems including roboticallycontrollable field generators that can be moved with a robotic arm ofthe system to facilitate field generator setup and instrument trackingwere discussed. In some of the examples, it was illustrated that the EMfield generator could be moved with the robotic arm to track an EMsensor, for example, to keep a moving EM sensor within a working volumeof the EM field generated by the EM field generator. By moving the EMfield generator, and correspondingly the working volume of the EM fieldthereof, the systems described herein can be considered to have an“expanded” working volume. That is, by repositioning the EM fieldgenerator and the working volume, the positions of EM sensors can bedetected and determined over an area that is larger than the workingvolume of a stationary field generator.

This section provides additional examples to elaborate on the concept of“expanding” the working volume of the EM field generator by moving theEM field generator with a robotic arm. In particular, the examples show,among other things, how EM sensors that are spaced too far apart to fitwithin the working volume of the EM field generator at the same time canbe tracked by moving the EM field generator, detecting the positions ofthe EM sensors, and mapping the detected positions of the EM sensorsinto the robotic coordinate frame. This provides significant benefitsover prior systems that utilize stationary field generators that may“lose track” of EM sensors once they are moved outside of the stationaryworking volume. The examples in this section with reference to theexample system 300 of FIGS. 23 and 29, but may be implemented with othersystems that include robotically controllable field generators as well.

A first example will now be described with reference to FIGS. 38A-38C,which illustrate the system 300 at different stages to show how thesystem 300 expand the working volume 306 of the EM field generator 302.As shown in FIG. 38A, which illustrates some components of the system ata first stage, the system includes the EM field generator 302 that isconfigured to generate an EM field having a working volume 306 withinwhich positions EM sensors 204 can be determined relative to an EMcoordinate frame 308, as previously described. The EM field generator302 is coupled to a robotic arm 212 that is configured to move to adjustthe position of the EM field generator 302. As the robotic arm 212 movesthe EM field generator 302, the position of the working volume 306 ofthe EM field generator 302 moves as well.

FIG. 38A also illustrates two EM sensors 204A and 204B. As shown, in thefirst stage, neither of the two EM sensors 204A, 204B are positionedwithin the working volume 306 of the field generator 302, and thus thepositions of the EM sensors 204A, 204B are not determinable in the firststage. Further, the two EM sensors 204A, 204B are positionedsufficiently far apart that both cannot be positioned within the workingvolume 306 at the same time. Thus, as will be shown in FIGS. 38B and38C, in order to track the positions of the EM sensors 204A, 204B, theEM field generator 302 can be moved, using the robotic arm 212, todifferent positions at which each of the EM sensors 204A, 204B can bedetermined and mapped into the robotic coordinate frame 216 in order to“expand” the working volume 306 of the EM field generator.

For example, the processor 380 of the system 300, which is incommunication with the EM field generator 302 and the robotic arm 212 towhich the EM field generator 302 is attached, can be configured to movethe EM field generator 302 to a first position at which the EM sensor204 is positioned within the working volume 306. With the EM fieldgenerator 302 in the first position, the processor 380 can determine theposition of the first EM sensor 204A within the working volume 306relative to the EM coordinate frame 308, and map the position of thefirst EM sensor 204A into the robotic coordinate frame 216 based on thekinematics of the robotic arm with the EM field generator 302 in thefirst position. The processor 380 can then cause the robotic arm 212 tomove the EM field generator 302 to a second position at which the secondEM sensor 204B is positioned within the working volume. And, with the EMfield generator 302 in the second position, the processor 380 candetermine the position of the second EM sensor 204B within the workingvolume 306 relative to the EM coordinate frame 308, and map the positionof the second EM sensor 204B into the robotic coordinate frame 216 basedon the kinematics of the robotic arm 212 with the EM field generator 302in the second position. In this way, the positions of both EM sensors204A, 204B can be presented within the robotic coordinate frame 216.

FIG. 38B illustrates the system in an example second stage, wherein theEM field generator 302 has been moved with the robotic arm 212 to thefirst position at which the first EM sensor 204A is positioned withinthe working volume 306. With the EM field generator 302 in thisposition, the position of the EM sensor 204A can be determined relativeto the EM coordinate frame 308, and that position can then be mapped tothe robotic coordinate frame 216 using the registration based on thekinematics of the robotic arm 212 previously described. In FIG. 38B, theEM sensor 204A has been shaded in black to indicate that its position,while initially determined in the EM coordinate frame 308, has beenmapped to the robotic coordinate frame 216.

FIG. 38C illustrate the system in an example third stage, wherein the EMfield generator 302 has been moved with the robotic arm 212 to thesecond position at which the second EM sensor 204B is positioned withinthe working volume 306. With the EM field generator 302 in thisposition, the position of the EM sensor 204B can be determined relativeto the EM coordinate frame 308, and that position can then be mapped tothe robotic coordinate frame 216 using the registration based on thekinematics of the robotic arm 212 previously described. As before, theEM sensor 204B has been shaded in black to indicate that its positionhas been mapped into the robotic coordinate frame 216. The EM sensor204A is also still shaded black to indicate its position within therobotic coordinate frame 216.

Considering FIG. 38C, one can see that the positions of both EM sensors204A, 204B can be simultaneously or substantially concurrentlyrepresented within the robotic coordinate frame 216, even though the EMsensors 204A, 204B are spaced sufficiently far apart that they bothcannot fit within the working volume 306 of the EM field generator 302at the same time. In some embodiments, the processor 380 is furtherconfigured to move the EM field generator 302 back and forth between thefirst position and the second position to frequently update (e.g., atabout 40 Hz, or time steps both smaller and larger) and track thepositions of the first and second EM sensors 204A, 204B.

Further, the example described with reference to FIGS. 38A-38C has beenpresented simplistically describing the EM field generator 302 in afirst position (FIG. 38B) and a second position (FIG. 38C). Inactuality, the system can be configured to continuously or nearcontinuously, detect any EM sensors 204 positioned within the EM fieldand map those positions into the robotic coordinate frame 216. Forexample, the processor 380 can be configured to move the EM fieldgenerator 302 using the robotic arm 212, and when an EM sensor isdetected within the working volume 306, the processor 380 can beconfigured to (i) determine the position of the EM sensor 204 within theworking volume 306 relative to the EM coordinate frame 308, and (ii) mapthe position of the EM sensor 204 into the robotic coordinate frame 216based on a kinematic pose of first robotic arm 212.

In some instances, the initial positions of the EM sensors 204 may notbe known. To locate the EM sensors 204, the system can be configured tomove the EM field generator 302 using the robotic arm 212 along asearching path or trajectory (e.g., along a predetermined path/patternor range of motions). The searching path can be configured to sweep theworking volume 306 through a treatment volume larger than the workingvolume 306 to locate EM sensors within the treatment volume. By sweepingthrough the treatment volume, the initial positions of any EM sensors204 within the treatment volume can be determined.

FIGS. 39A-39D illustrate an embodiment of the robotic medical systemincluding a robotically controllable field generator 302 that can bemoved with the robotic arm 212 along a searching path 510 to detectpositions of EM sensors 204. In the example of FIG. 39A, the initialpositions of two EM sensors 204A, 204B are not known. In an effort toidentify the positions of the EM sensors 204A, 204B, FIG. 39Aillustrates an example searching path 510 along which the EM fieldgenerator 302 will be moved using the robotic arm 212. In theillustrated embodiment, the searching path 510 comprises a generallysinusoidal path configured to sweep the working volume 306 across thetreatment volume or treatment site in which EM sensors 204 are expected.The searching path 510 illustrated in FIG. 39A is provided by way ofexample, and any number of paths comprising different shapes arepossible.

FIG. 39B illustrates the system at a second stage, as the EM fieldgenerator 302 is moved along the searching path 510. At the second stageillustrated in FIG. 39B, the first EM sensor 204A is now positioned anddetectable within the working volume 306 of the EM field generator 302.The position of the EM sensor 204A can be determined relative to the EMcoordinate frame 308, and that position can then be mapped to therobotic coordinate frame 216 using the registration based on thekinematics of the robotic arm 212 previously described. In FIG. 39B, theEM sensor 204A has been shaded in black to indicate that its position,while initially determined in the EM coordinate frame 308, has beenmapped to the robotic coordinate frame 216. The EM field generator 302can continue along the searching path 510.

FIG. 39C illustrates the system at a third stage, as the EM fieldgenerator 302 is moved even further along the searching path 510. At thethird stage illustrated in FIG. 39C, the second EM sensor 204B is nowpositioned and detectable within the working volume 306 of the EM fieldgenerator 302. The position of the EM sensor 204B can be determinedrelative to the EM coordinate frame 308, and that position can then bemapped to the robotic coordinate frame 216 using the registration basedon the kinematics of the robotic arm 212 previously described. In FIG.39B, the EM sensor 204B has been shaded in black to indicate that it hasbeen mapped to the robotic coordinate frame 216. The EM field generator302 can continue along the searching path 510. The position of the EMsensor 204A is also shown, shaded in black, to represent its positionmapped into the robotic coordinate frame.

FIG. 39D illustrates the system at a fourth stage at which the EM fieldgenerator 302 has completed the searching path 510. As show, thepositions of all EM sensors 204 (in the illustrated example, the EMsensors 204A, 204B) have been identified and mapped into the roboticcoordinate frame 216. In some embodiments, the system is configured tofollow a searching path 510 at the beginning of the procedure toidentify the positions of EM sensors 204. The searching path 510 can beperiodically repeated to determine whether any new EM sensors 204 havebeen introduced.

In some embodiments, the initial positions of the EM sensors 204 can bedetermined in other ways. For example, when an EM sensor 204 is includedon a robotically controllable instrument, the system may estimate ordetermine the initial position of the EM sensor 204 based on thekinematics of the robot. For EM sensors 204 that are not coupled torobotic components of the system, in some embodiments, a user mayindicate or input the initial positions of the EM sensors 204 to thesystem (e.g., via a user input device, controller, graphical userinterface, etc.). For example, in the case of an EM patch sensor placedon a patient's chest to track respiration, the user placing the EM patchsensor may indicate where the patch sensor has been placed to thesystem.

As noted above, in some instances, the mapped positions of the EMsensors 204 into the robotic coordinate frame 216 may not represent livepositions of the sensors. Accordingly, after the initial positions ofthe EM sensors are determined, the system may determine a new trackingpath along which the EM field generator 302 can be moved in order tocontinue tracking the positions of the EM sensors 204. The tracking pathmay, for example, be shorter than the searching path, which can allowfor faster or more frequent remapping of the positions of the EM sensors204. An example is described with reference to FIGS. 40A-40D.

FIGS. 40A-40D continue the example of FIGS. 39A-39D and furtherillustrate an example tracking path 512 for the EM field generator 302.The tracking path 512 can be determined based on the determinedpositions of the EM sensors 204 within the robotic coordinate frame 216.For example, the tracking path 512 can be determined such that the EMfield generator 302 is moved back and forth between determined positionsof the EM sensors 204 within the robotic coordinate frame 216 so thatthe positions can be frequently remapped and updated frequently. Forexample, as the EM field generator 302 is moved along the tracking pathusing the robotic arm 212, the processor 380 can be configured to (i)re-determine the positions of the EM sensors 204 within the workingvolume 306 relative to the EM coordinate frame 308, (ii) re-map thepositions of the EM sensors 204 into the robotic coordinate frame 216based on the kinematic pose of the robotic arm 212, and (iii) determinean updated tracking path 510.

As shown in FIG. 40A, the tracking path 512 can be shorter than thesearching path 510 (of FIGS. 39A-39D). This can be advantageous as itcan limit unnecessary movement of the EM field generator (which mightcause collisions with other objects in the space) and limits the time atwhich no EM sensors are positioned within the working volume 306. In theillustrated embodiment, the tracking path 512 comprises a line thatmoves the EM field generator 302 back and forth between positions atwhich each of the EM sensors 204A, 204B can be detected within theworking volume 306 and remapped into the robotic coordinate frame 216.Other shapes for the tracking path 512 are also possible.

FIG. 40A illustrates the EM field generator 302 positioned at a firstend of the tracking path 512 at which the first EM sensor 204A ispositioned within the working volume 306. In this position, the positionof the first EM sensor 204A can be detected relative to the EMcoordinate frame 308 and remapped into the robotic coordinate frame 216.FIG. 40B illustrates the EM field generator 302 positioned at a secondend of the tracking path 512 at which the second EM sensor 204B ispositioned within the working volume 306. In this position, the positionof the second EM sensor 204B can be detected relative to the EMcoordinate frame 308 and remapped into the robotic coordinate frame 216.In FIGS. 40A and 40B, the EM sensors 204A, 204B are shaded in black torepresent that their positions have been mapped into the roboticcoordinate frame 216 (e.g., when detected as the EM field generator 302moves along the searching path 510 as described in the previous exampleof FIGS. 39A-39D).

FIG. 40C illustrates how the mapped position of the EM sensor 204A intothe robotic coordinate frame may not be live. For example, as shown inFIG. 40C, while the EM field generator 302 is positioned at the secondend of the tracking path 512 such that the first EM sensor 204A is notpositioned within the working volume 306, the first EM sensor 204A canbe moved to a new position indicated as EM sensor 204A′. Notably, theprevious mapped position of the EM sensor 204A is still shown (shaded inblack). In this position, the system has not yet determined that the EMsensor 204A has moved to the new position of the EM sensor 204A′. Insome embodiments, each sensor 204 is connected to a dedicated port ofthe system, such that the system can identify and distinguish betweenthe sensors. Other mechanisms and methods for distinguishing between thesensors are also possible as well.

However, FIG. 40D illustrates that, as the EM field generator 302 ismoved back to along the tracking path 512, it can detect the newposition of the EM sensor 204A′. In this position, the new position ofthe first EM sensor 204A′ can be detected relative to the EM coordinateframe 308 and remapped into the robotic coordinate frame 216. Toindicate that the new position of the EM sensor 204A′ has been remappedto the robotic coordinate frame 216, it has been shade in black in FIG.40D.

FIG. 40D also illustrates that a new tracking path 512′ can bedetermined based on the new position of the EM sensor 204A′.Accordingly, as EM sensors 204 are moved, the new positions of the EMsensors 204 can be remapped into the robotic coordinate frame 216 and anupdated tracking path 512′ can be determined. This process can repeatthroughout the procedure to beneficially allow tracking of various EMsensors 204 within an expanded working volume that is made possible bymoving the EM field generator 302 with the robotic arm 212 and mappingthe positions of detected EM sensors 204 into the robotic coordinateframe based on the kinematic pose of the robotic arm 212.

FIG. 41A is a flowchart illustrating an example method 600 for expandinga working volume of a robotically controllable field generator. Themethod 600 can be used, for example, to implement the functionalitydescribed above with reference to FIGS. 38A-38C. The method 600 beginsat block 601, at which the EM field generator is moved to a firstposition. The first position can be a position at which an EM sensor isdetected within the working volume of the EM field generator. The EMfield generator can be moved using the robotic arm to which it isattached. In some embodiments, moving the EM field generator to thefirst position comprises moving the EM field generator along a searchingor tracking path.

The method proceeds to block 602. At block 602, the position of thefirst EM sensor is determined relative to the EM coordinate frame. TheEM coordinate frame is associated with the EM field of the EM fieldgenerator. Next, at block 603, the position of the EM sensor within theEM coordinate frame (determined at block 602) is mapped to a roboticcoordinate frame. The robotic coordinate frame is associated with, forexample, the robotic arm to which the EM field generator is coupled andthe other robotic components of the system. Mapping the position of theEM sensor into the robotic coordinate frame can be based on thekinematic pose of the first robotic arm, which as noted previously,establishes a relationship between the EM coordinate frame and therobotic coordinate frame since the EM field generator is coupled to therobotic arm.

The method 600 can then proceed to block 604. At block 604, the EM fieldgenerator is moved to a second position at which a second EM sensor isdetected within the working volume. As before, the EM field generatorcan be moved to the second position using the robotic arm to which it isattached. At block 605, the position of the second EM sensor can bedetermined relative to the EM coordinate frame. Next, at block 606, theposition of the second EM sensor within the EM coordinate frame(determined at block 605) is mapped to a robotic coordinate frame, againusing the registration based on the kinematics of the robotic arm.Finally, line 607 illustrates that the method 600 can be performed as aloop to continually re-detect and re-map the positions of the first andsecond EM sensors into the robotic coordinate frame.

FIG. 41B is a flowchart illustrating another example method 610 forexpanding a working volume of a robotically controllable fieldgenerator. The method 610 begins at block 611, which involves moving anEM field generator with a robotic arm to which it is attached. At block612, whenever an EM sensor is detected within the working volume of theEM field generator, the position of the EM sensor can be determinedrelative to the EM coordinate frame. At block 613, the position of theEM sensor can be mapped to the robotic coordinate frame. The mapping canbe based on the kinematics of the robotic arm as described previously.Finally, line 614 illustrates that the method 610 can be performed as aloop to continually re-detect and re-map the positions of the first andsecond EM sensors into the robotic coordinate frame.

iii. Alignment of Percutaneously Insertable Instrument with an EM Target

The robotically controllable EM field generator 302 may also facilitatealignment of percutaneously inserted tools with other toolsendoscopically inserted into the body. For example, as shown in FIG.27B, the ureteroscope 502 can be guided endoscopically to a positionwithin the kidney. The operator may then desire to make a percutaneousinsertion of or with a needle (or other instrument) so as to rendezvouswith the ureteroscope 502. In some embodiments, the EM field generator302 can include a needle guide (e.g., a channel, tube, or otherstructure) through which a needle (or other instrument) can be inserted.In other embodiment, the EM field generator 302 can define a spacethrough which the needle guide or the like may be advanced and/orpositioned. For example, the needle guide can be configured such that itallows for only one degree of freedom (insertion and/or retraction) forneedles inserted therethrough. The EM field generator 302 can be alignwith the EM sensor 204 on the ureteroscope, such that when the needle isinserted through the needle guide, it is guided directly toward the EMsensor 204. One such use-case is in percutaneous procedures where accessto the inner organs or other tissue is done via needle-puncture of theskin. A needle that is rigidly attached to the robot end effector andhas a known transformation (rigid-body-definition) with respect to therobot end effector, thus can be robotically aligned with an EM beacon(target) inside the patient's body. The needle can then be insertedthrough the skin along the robotically-aligned trajectory.

FIGS. 42A-42C illustrate an embodiment of a robotic medical system, suchas the system 300 of FIGS. 23 and 29, with a robotically controllablefield generator 302 configured to facilitate alignment of apercutaneously insertable instrument 214 with an EM target 204A. Thepercutaneously insertable instrument 214 can comprise, for example, aneedle, an access sheath, a laparoscopic instrument, or other type ofpercutaneously insertable instrument. In the example of FIGS. 42A-42C itmay be desired to precisely insert the percutaneously insertableinstrument 214 into the body at a specific location. To facilitatepercutaneous insertion, an EM target 204A (which can be an EM sensor)can be placed within the body at the desired position for thepercutaneously insertable instrument 214. In some embodiments, thepercutaneously insertable instrument 214 can also include an EM sensor204B positioned thereon.

In the illustrated embodiment of FIG. 42A, the percutaneously insertableinstrument 214 can be a robotically controllable instrument. Thepercutaneously insertable instrument 214 is shown attached to a secondrobotic arm 212B that is configured to position and insert thepercutaneously insertable instrument 214. As shown, the percutaneouslyinsertable instrument 214 extends along an axis 502. Generally, it isdesirable to insert the percutaneously insertable instrument 214 alongthe axis 502. However, in order to insert the percutaneously insertableinstrument 214 accurately, the axis 502 must be aligned with the EMtarget 204A.

Accordingly, the processor 380 of the system can be configured todetermine a registration that maps positions within an EM coordinateframe associated with the EM field generator 302 to positions within arobotic coordinate frame 216 based on a kinematic pose of the firstrobotic arm 212A, to which the EM field generator 302 is attached. Theprocessor 380 can further be configured to determine, based on theregistration, a position of the EM target 204A within the roboticcoordinate frame 216, and, based on the position of the EM target 204Awithin the robotic coordinate frame 216, the processor can be configuredto move the second robotic arm 212B to align the axis 502 of thepercutaneously insertable instrument 214 with the EM target Finally, theprocessor 380 can be configured to insertion of the percutaneouslyinsertable instrument 214 along the axis 502 towards the EM target 204Ausing the second robotic arm 212B.

As described previously, the registration can be determined based on theposition of the EM field generator 302 within the robotic coordinateframe 216, and the position of the EM field generator with the roboticcoordinate frame 216 can be determined based on the kinematic pose ofthe first robotic arm 212A. The processor 380 can further be configuredto determine the position of the EM target 204A within the roboticcoordinate frame 216 by determining a position of the EM target 204Awithin the EM coordinate frame 308, and mapping, using the registration,the position of the EM target 204A within the EM coordinate frame 308 tothe position of the EM target 204A within the robotic coordinate frame216.

In some embodiments, the EM target 204A can be an EM sensor positionedon another robotic medical instrument configured for insertion into thepatient, such as an endoscope navigated through the body. This roboticmedical instrument can be coupled to a third robotic arm that isconfigured to control the robotic medical instrument;

FIG. 42A illustrates the system in a first state, wherein the axis 502of the percutaneously insertable instrument 214 is not aligned with theEM target 204A. FIG. 42B illustrates the system in a second state, afteralignment of the axis 502 with the EM target 204A. FIG. 42C illustratesthe system in a third state, showing insertion of the percutaneouslyinsertable instrument 214 along the axis 502 and rendezvous with the EMtarget 204A.

FIGS. 43A and 43B illustrate an embodiment of a robotic medical system,such as the system 300 of FIGS. 23 and 29, with a roboticallycontrollable field generator 302 configured to facilitate alignment ofan instrument guide 520 mounted on the field generator 302 with an EMtarget 204A. The instrument guide 520 can be configured such that apercutaneously insertable instrument can be inserted therethrough alongan axis 522 during insertion into the patient. The instrument guide 520can limit motion of the percutaneously insertable instrument to motiononly along the axis 522, such that when the axis 522 is aligned with theEM target 204, the percutaneously insertable instrument can be guidedtoward the EM target 204. In some embodiments, the instrument guide 520is removably coupled to the EM field generator 302. In otherembodiments, the instrument guide 520 can be permanently coupled to orintegrated into the EM field generator 302. In related aspects, it isnoted that the instrument guide 520 may be configured to haveshapes/curves and/or lengths different than the example of FIGS. 43A and43B.

Similar to the example previously described, the axis 522 of theinstrument guide 520 can be aligned with the EM target 204 to guide apercutaneously insertable instrument toward the EM target 20A. Forexample, the processor 380 can be configured to determine a registrationthat maps positions within an EM coordinate frame 308 associated withthe EM field generator 308 to positions within a robotic coordinateframe 216 based on a kinematic pose of the first robotic arm 212. Theprocessor 380 can also determine, based on the registration, a positionof the EM target 204 within the robotic coordinate frame 216, and, basedon the position of the EM target 204 within the robotic coordinate frame216, the processor 280 can cause movement of the first robotic arm 212to align the insertion axis 522 of the instrument guide 520 with the EMtarget 204.

FIG. 43A illustrates the system in a first state, wherein the axis 522of the instrument guide 520 is not aligned with the EM target 204. FIG.43B illustrates the system in a second state, after alignment of theaxis 522 with the EM target 204. With the system in the second state, apercutaneously insertable instrument can be inserted through theinstrument guide 520 along the axis 522 toward the EM target 204.

The system may also include the percutaneously insertable instrument.The percutaneously insertable instrument can comprise a needle, anaccess sheath, a laparoscopic instrument or another type ofpercutaneously insertable instrument. The percutaneously insertableinstrument can extends along an axis. In some embodiments, the systemcomprises a second robotic arm coupled to and configured to move thepercutaneously insertable instrument. In such cases, the processor canfurther be configured to align the axis of the percutaneously insertableinstrument with the insertion axis using the second robotic arm, andinsert the percutaneously insertable through the instrument guide alongthe insertion axis toward the EM target 204 using the second roboticarm. In some embodiments, the percutaneously insertable instrument maycomprise an instrument manually inserted through the instrument guide520.

In some embodiments, the EM target 204 comprises an EM sensor positionedon a medical instrument configured for insertion into the patient. Therobotic medical instrument can be an endoscope. The system may comprisea third robotic arm coupled to and configured to control the roboticmedical instrument

FIGS. 44A and 44B illustrate an embodiment of a robotic medical systemwith a robotically controllable field generator 302 configured tofacilitate alignment of an instrument guide 530 with an EM target 204.This example is similar to the example of FIGS. 43A and 43B, except thatthe instrument guide 530 is coupled to a second robotic arm 212B, ratherthan to the EM field generator 302. In a similar manner, the processor380 can be configured to determine a registration that maps positionswithin an EM coordinate frame 308 associated with the EM field generator302 to positions within a robotic coordinate frame 216 based on akinematic pose of the first robotic arm 212A, to which the EM fieldgenerator 302 is attached. The processor 380 can further determine,based on the registration, a position of the EM target 204 within therobotic coordinate frame, and based on the position of the EM targetwithin the robotic coordinate frame, move the second robotic arm 212B toalign an insertion axis 532 of the instrument guide with the EM target204. FIG. 44A illustrates the system prior to alignment of the axis 532of the instrument guide 530 with the EM target 204, and FIG. 44Billustrates the system after alignment. In the position illustrated ofFIG. 44B, a percutaneously insertable instrument can be inserted throughthe instrument guide 530 toward the EM target 204.

In some embodiments, the percutaneously insertable instrument is coupledto a third robotic arm that is configured to move the percutaneouslyinsertable instrument. In such cases, the processor can be furtherconfigured to align the axis of the percutaneously insertable instrumentwith the insertion axis 532 of the instrument guide 530 using the thirdrobotic arm, and insert the percutaneously insertable through theinstrument guide 530 along the insertion axis 532 toward the EM target204 using the third robotic arm. In other embodiments, thepercutaneously insertable instrument can be manually inserted throughthe instrument guide 530.

The system may also comprise a robotic medical instrument configured forinsertion into the patient. The EM target 204 can comprise an EM sensoron the robotic medical instrument. The robotic medical instrument can becoupled to another robotic arm that is configured to control the roboticmedical instrument.

FIG. 45A is a flowchart illustrating a method 700 for aligning apercutaneously insertable instrument with an EM target using arobotically controllable field generator. The method 700 begins at block701, at which a registration that maps positions within an EM coordinateframe to positions within a robotic coordinate frame. The registrationcan be determined based on a kinematic pose of a robotic arm to which anEM field generator is attached as previously described.

Next, at block 702, the position of an EM target can be determined inthe robotic coordinate frame based on the registration. As describedpreviously, this can involve determining the position of the EM targetwithin an EM field generated by the EM field generator and mapping thatposition into the robotic coordinate frame.

Based on the determined position of the EM target, at block 703, an axisof a percutaneously insertable instrument is aligned with the EM target.An example is shown in FIG. 42B, described above. Finally, at block 704,the percutaneously insertable instrument can be inserted along the axistowards the EM target. An example is shown in FIG. 42C, describedpreviously.

FIG. 45B is a flowchart illustrating a method 710 for aligning aninstrument guide for a percutaneously insertable instrument with an EMtarget using a robotically controllable field generator. At block 711,the method 700 involves determining a registration that maps positionswithin an EM coordinate frame to positions within a robotic coordinateframe. At block 712, the position of an EM target is determined withinthe robotic coordinate frame based on the registration. And finally, atblock 713, the method 700 involves aligning an axis of the instrumentguide with the EM target. Examples have been described above withreference to FIGS. 43A and 43B, which show an instrument guide on the EMfield generator, and FIGS. 44A and 44B, which show an instrument guidemounted on a second robotic arm.

iv. Distortion Detection

The robotically controlled EM field generator 302 can also be used fordistortion detection. For example, a robotically controlled EM fieldgenerator can be commanded to move while a static EM sensor is presentin the region of interest, e.g., a needle-insertion site, a biopsy site,or any static position within the working volume of the EM fieldgenerator. The EM field generator 302 can be moved by a robotic arm 212to which it is attached and the commanded robot motion (within a roboticcoordinate frame) can be compared with the recorded EM sensor trajectory(within an EM coordinate frame). The difference between the twotrajectories can be a measure or indicator of the distortion in the EMsignal within the working volume of the EM field generator 302. Asimilar principle can also be used with a non-static EM sensor whosemotion is either known or has bounded uncertainty. The accuracy ofdistortion detection may depend on or be bounded by the uncertainty ofthe knowledge of the EM sensor motion/location.

FIG. 46A, for example, illustrates an embodiment of a robotic medicalsystem moving an EM field generator 302 relative to a fixed EM positionsensor 204 in order to detect EM distortion. As shown in the illustratedembodiment, the EM field generator 302 can be coupled to a first roboticarm 212. The EM field generator 302 can be configured to generate an EMfield and is associated with an EM coordinate frame within which theposition of EM sensors, such as the illustrated EM sensor 204 can bedetermined with reference to the EM coordinate frame. The first roboticarm 212 can be configured to move to reposition the EM field generator302. The first robotic arm 212 can be associated with a roboticcoordinate frame. As described above, the motion of the robotic arm 212within the robotic coordinate frame is known due to the known kinematicsof the arm. Accordingly, since the EM field generator 302 is coupled tothe first robotic arm 212, the position of the EM field generator 302within the robotic coordinate frame 212 can also be determined based onthe kinematic pose of the arm. As described in more detail above, thisrelationship can establish a registration that can be used to determinepositions of EM sensors 204 within the EM coordinate frame of the EMfield with respect to the robotic coordinate frame.

In the illustrated embodiment of FIG. 46A, in order to determine ordetect EM distortion within the EM field, the EM sensor 204 can beprovided at a fixed and stationary position. For example, the EM sensor204 can be affixed to a non-moving object, such as a patient platform orother non-moving piece of equipment. In some embodiments, the EM sensor204 is positioned on a medical instrument, such as a roboticallycontrolled medical instrument. If this is the case, during a distortiondetection step as described with reference to FIGS. 46A-47, the medicalinstrument can remain stationary such that the position of the EM sensor204 remains fixed and stationary.

In order to detect EM distortion, a processor in communication with thefirst robotic arm 212 (and, by extension, the EM field generator 302)can be configured to cause the first robotic arm 212 to move the EMfield generator 302 along a robotic trajectory 850. In FIG. 46A, therobotic trajectory 850 is represented by a dashed line, and the systemis illustrated approximately halfway through the movement, such that theEM field generator 302 is positioned about halfway along the robotictrajectory 850. In some embodiments, movement of the EM field generator302 by the first robotic arm 212 along the robotic trajectory 850 can beaccomplished under direction or control of an operator, such as aphysician, who can command the motion using a controller or other userinterface. In other embodiments, the movement of the EM field generator302 by the first robotic arm 212 along the robotic trajectory 850 can beautomatically provided, for example, as part of an automatic EMdistortion detection process operated by the system.

In FIG. 46A, it is important to realize that, in this example, the EMfield generator 302 is moved along the robotic trajectory 850, and thatthis occurs while the EM sensor 204 remains stationary or fixed asdescribed above. FIG. 46A also illustrates an EM sensor trajectory 852,which is represented in the figure as a dot-dash lined. The EM sensortrajectory 852 is, however, not a result of physical movement of the EMsensor 204, which as noted above, remains fixed. Rather, the EM sensortrajectory 852 is generated due to the movement of the EM fieldgenerator 302 relative to the stationary EM sensor 204. For example, asthe first robotic arm 212 moves the EM field generator 302 along therobotic trajectory 850, the position of the EM sensor within the EMcoordinate frame associated with the EM field generator 302 is recorded,producing the EM trajectory 852.

Accordingly, the robotic trajectory 850 is caused by the movement of theEM field generator 302 by the first robotic arm 212. The robotictrajectory 850 can be detected, determined, and/or recorded based on theknown kinematic movement of the first robotic arm 212 during the motion.In contrast, the EM sensor trajectory 852 is caused by the movement ofthe EM field generator 302 relative to the stationary EM sensor 204. TheEM sensor trajectory 852 can be detected, determined, and/or recordedbased on detecting the EM sensor 204 within the EM field of the EM fieldgenerator 302 as the EM field generator 302 is moved and the EM sensor204 remains stationary.

Because the EM sensor 204 remains fixed during movement of the EM fieldgenerator 302 by the first robotic arm 212, in the absence of any EMdistortion, one would expect the robotic trajectory 850 to correspond tothe EM sensor trajectory 852. This is because the movement of the EMfield generator 302 should match the relative motion between the movingEM field generator 302 and the static EM sensor 204. Accordingly,differences between the robotic trajectory 850 and the EM sensortrajectory 852 can be analyzed to determine whether EM distortion ispresent. In some embodiments, the analysis may also provide a measure ofthe degree or severity of the EM distortion.

FIG. 46B illustrates comparison of the robotic trajectory 850 and the EMsensor trajectory 852. In this example, start points of each trajectoryhave been aligned, although this need not be necessary in allembodiments. As shown in FIG. 46B, it is apparent that the robotictrajectory 850 and the EM sensor trajectory 852 do not exactlycorrespond. The difference(s) between the trajectories can be determinedin order to detect EM distortion. In some embodiments, the detected EMdistortion is indicative of EM distortion at the location of the fixedEM sensor 204.

Not all differences between the robotic trajectory 850 and the EM sensortrajectory 852 are necessarily caused by EM distortion. For example, therobotic system may not be able to determine robotic motion with totalaccuracy for various reasons including manufacturing tolerances, motorcontrol limitations, sensor accuracy, etc. Accordingly, some of thedifferences between the robotic trajectory 850 and the EM sensortrajectory 852 may be caused by imprecision in determining roboticmotion. In general, however, imprecision in determining robotic motioncan be small, for example, in a sub-millimeter range. Similarly, the EMfield generator 302 may be subject to EM noise that can lead to areduced resolution of EM sensor positions determined within the field.This can lead to imprecise recording of the EM sensor trajectory 852,causing differences, that are not directly attributable to EMdistortion. Again, however, an EM field generator can be configured orselected such that the EM sensor noise is, in some embodiments, around 1millimeter or less. Accordingly, for some embodiments of the roboticsystem, imprecision in the determination of the robotic motion and alsoimprecision in the recording the EM sensor position within the EM fieldcan lead to potential inaccuracies of about 1 millimeter or less.

Accordingly, to determine the presence of EM distortion, it can bebeneficial to analyze differences between the robotic trajectory 850 andthe EM sensor trajectory 852 with respect to a threshold value that isselected to exclude or reduce the contributions due to other factors. Insome embodiments, the threshold value can be, for example, 1 millimeter,1.25 millimeters, 1.5 millimeters, 1.75 millimeters, 2 millimeters, 2.25millimeters, 2.5 millimeters, 2.75 millimeters, 3 millimeters orgreater. In many instances, it is beneficial to set the threshold valuesuch that it is larger than one or both of an error factor associatedwith motion of the first robotic arm or an error factor associated withEM sensor or generator noise.

Comparing the robotic trajectory 850 and the EM sensor trajectory 852can be accomplished in a variety of ways. For example, the shapes of therobotic trajectory 850 and the EM sensor trajectory 852 can be directlycompared as shown in FIG. 46B. The shapes can be analyzed to determinehow closely they correspond, and if the deviation between the shapesexceeds a threshold, the system can determine that EM distortion ispresent and/or the degree of EM distortion. In another embodiment,analyzing the robotic trajectory 850 and the EM sensor trajectory 852 todetermine a difference therebetween can comprise comparing a pluralityof points along the robotic trajectory 850 to a corresponding pluralityof points along the EM sensor trajectory 852. In some embodiments, theplurality of points are determined with respect to a time associatedwith movement of the EM field generator 302 using the first robotic arm212. For example, a point on the robotic trajectory 850 at the start ofthe movement (e.g., at t=0) can be compared with a corresponding pointon the EM sensor trajectory 852. Then, subsequent points can be comparedat discrete time steps thereafter.

In some embodiments, if EM distortion is detected, the system can beconfigured to reposition the EM field generator 302 with the firstrobotic arm 212 at a location that reduces the EM distortion. Forexample, in some embodiments, the system can position the EM fieldgenerator 302 at a position along the robotic trajectory 850 at which adifference between the robotic trajectory 850 and the EM sensortrajectory 852 is reduced.

In some embodiments, this can be achieved by moving the EM fieldgenerator 302 along a second robotic trajectory in an effort to identifya location that can reduce the EM distortion. Accordingly, the EM fieldgenerator 302 can be moved along a second robotic trajectory that isdifferent than the first robotic trajectory 850. For example, the EMfield generator 302 can be moved along the second robotic trajectorythrough different space than was covered along the first robotictrajectory 850. As before, this is done while the EM sensor 204 remainsstationary. The second robotic trajectory can be recorded along with asecond and corresponding EM sensor trajectory. These trajectories can beanalyzed to determine if a location that reduces EM distortion can bedetermined. If so, the EM field generator 302 can be moved, using therobotic arm 212, to that position. If not, additional robotictrajectories can be attempted until a suitable location for the EM fieldgenerator 302 that reduces EM distortion can be found.

In the examples described above with respect to FIGS. 46A and 46B, therobotic trajectory 850 has been defined with respect to a global robotreference frame, whereas the EM sensor trajectory 852 is defined withrespect to the EM field generator coordinate frame. In this case, inorder to compare the robotic trajectory 850 to the EM sensor trajectory852, the EM field generator coordinate frame must be registered to theglobal robot reference frame. This can be achieved, for example, asdescribed above, because the EM field generator 302 is attached to therobotic arm 212, establishing a kinematic relationship between theglobal robot reference frame and the EM field generator coordinateframe.

Moreover, the examples described above with respect to FIGS. 46A and 46Bare somewhat simplified, for ease of understanding and illustration,with the assumption that only the position of the EM field generator 302relative to the EM sensor 204 has been changed and not the relativeorientation. In more complex examples, both the position and orientationof the EM field generator 302 relative to the EM sensor 204 can bechanged and the detected position (and/or orientation) of the EM sensor204 can be analyzed to determine whether EM distortion is present(and/or a degree thereof). In such cases, the determination of EMdistortion can be made with respect to a global reference frame 854associated with a cart to which the robotic arm 212 is attached, asshown in FIG. 46C, which illustrates the robotic arm 212 to which the EMfield generator 302 is attached as it moves along a robotic trajectory850 that changes the position and orientation of the EM field generator302 relative to the EM sensor 204. Because the cart and the EM sensor204 are not moving, any detected position and/or orientation changemeasured with respect to the global reference frame 854 would be due todistortion (and negligible kinematics error).

EM distortion can be caused by any or all components that are positionedwithin the working volume of the EM field generator. Accordingly,similar processes can be performed while moving other components in aneffort to find optimal or preferred positions for other components.

FIG. 47 is a flowchart illustrating an example method 800 for EMdistortion detection. The method 800 is similar in many respects to theprocess described above with respect to FIGS. 46A 46C. The method 800can be performed using a robotic medical system, such as the systemspreviously described, in which an EM field generally is roboticallycontrollable via attachment to a robotic arm of the system.

The method 800 begins at block 801. Block 801 can involve moving an EMfield generator that is coupled to a first robotic arm with the firstrobotic arm along a robotic trajectory while an EM sensor remainsstationary at a location. As described above, the robotic trajectory canbe determined based on the kinematics of the arm to which the EM fieldgenerator is attached. In some embodiments, the robotic trajectory canbe determined with respect to a global reference frame, which may, forexample, be associated with a cart to which the robotic arm is attached.In some embodiments, block 801 is performed by a processor (orprocessors) of a robotic medical system. For example, a processor may beconfigured to transmit a command to the first robotic arm to causemovement of the EM field generator along a robotic trajectory. In somecases, the EM sensor may be fixed to determinable position while the EMfield generator moves along the robotic trajectory.

At block 802, an EM sensor trajectory of the EM sensor within an EMfield associated with the EM field generator can be detected based onthe sensor data generated by the EM sensors as the EM field generatormoves along the robotic trajectory. In this way, and as notedpreviously, the EM trajectory may be the recorded position of the EMsensor within the EM field during the same time period as when the EMfield generator was moved by the robotic arm according to the robotictrajectory.

At block 803, the robotic trajectory and the EM sensor trajectory areanalyzed to determine difference(s) therebetween. Analysis can beperformed in a number of ways, including comparing the shapes of thetrajectories and/or comparing a plurality of discrete points along thetrajectories, among others. Such comparison can consider changes inposition and/or orientation.

At block 804, EM distortion can be detected at the location of the EMsensor based on a comparison between the difference between thetrajectories and a threshold. The threshold can be selected ordetermined so as to reduce or exclude contributions caused by factorsother than EM distortion, such as errors in recording or determiningrobotic motion and/or in detecting the EM sensor position within the EMfield.

FIG. 48 is a flowchart illustrating another example method 900 for EMdistortion detection. The method 900 can be performed using a roboticmedical system, such as the systems previously described, in which an EMfield generally is robotically controllable via attachment to a roboticarm of the system. The method 900 begins at block 901, at which a firstrobotic arm is coupled to an EM field generator. The EM field generatoris configured to generate an EM field and is associated with an EMcoordinate frame. The first robotic arm is associated with a roboticcoordinate frame and the position of the coupled EM field generator canbe determined within the robotic coordinate frame based on the kinematicpose of the first robotic arm. In some embodiments, the position canfurther be determined with respect to a global coordinate frame, such asa coordinate frame associated with a cart to which the robotic arm isattached. A position of an EM sensor within a working volume of the EMfield of the EM field generator can be determined and mapped into therobotic coordinate frame (or the global coordinate frame) based on thepose of the arm as previously described.

At block 902, with the EM field generator at a first EM field generatorposition relative to the EM sensor, the method 900 includes determininga first registration between the EM coordinate frame and the roboticcoordinate frame based on the kinematics of the first robotic arm withthe EM field generator at the first position and determine a first EMsensor position of the EM sensor within the robotic coordinate framebased on the first registration.

At block 903, the method 900 includes moving the EM field generator fromthe first EM field generator position to a second EM field generatorposition relative to the EM sensor using the first robotic arm, whereinthe EM sensor remains stationary during the movement. In someembodiments, movement of the EM field generator by the first robotic armcan be accomplished under direction or control of an operator, such as aphysician, who can command the motion using a controller or other userinterface. In other embodiments, the movement of the EM field generatorby the first robotic arm along the robotic trajectory can beautomatically provided, for example, as part of an automatic EMdistortion detection process operated by the system.

At block 904, with the EM field generator at the second EM fieldgenerator position, a second registration between the EM coordinateframe and the robotic coordinate frame based on the kinematics of thefirst robotic arm with the EM field generator at the second EM fieldgenerator position is determined, and a second EM sensor position of theEM sensor within the robotic coordinate frame (or the global coordinateframe) based on the second registration is also determined. The secondregistration is necessary because the EM field generator has been moved,the new kinematic pose of the robotic arm at the second position is usedto establish the new relationship between the robotic coordinate frameand the EM coordinate frame.

At block 905, the method 900 includes determining a difference betweenthe first EM sensor position and the second EM sensor position. Sincethe EM sensor has not moved, one would expect the determined positionsto be the same. However, EM distortion caused by the different positionsof the robotic arm and EM field generator (or other components that havebeen moved) can cause the determined positions to vary. In someembodiments, changes in orientation may also be considered. At block906, EM distortion can be detected based on a comparison between thedifference and a threshold (e.g., the difference exceeding thethreshold). As noted above, other factors can contribute to thedifference. As before, the threshold can be selected or determined toreduce the contribution of these other, non-EM distortion, factors.

When viewed relative to the threshold, a difference exceeding thethreshold can indicate that EM distortion is present. However, by onlytesting two EM field generator positions (and/or orientations(, it maybe difficult to determine which position provided the most accuratedetermination of position. Accordingly, it can be advantageous tocontinue to test a plurality of other EM field generator positions(and/or orientations) by repeating the associated steps of:

moving the EM field generator to a new position using the robotic arm,determining a new EM to robotic coordinate registration based on the newposition of the EM field generator based on the kinematics of therobotic arm, and determining the position of the EM sensor within therobotic coordinate frame based on the newly determined registration.

In some embodiments, this process can be continued to build a map of theEM distortion by determining the distortion at a plurality of locations.Locations of the EM field generator that that provide determinations ofthe EM sensor position with the least variation from the other positionscan be determined to have low EM distortion, while locations of the EMfield generator that that provide determinations of the EM sensorposition with the most variation from the other positions can bedetermined to have high EM distortion. In this way, the system or anoperator can determine how best to position the EM field generator andother components so as to reduce EM distortion.

EM distortion that is detected using the methods described above (aswell as EM distortion generally) can be both static or dynamic. StaticEM distortion is unchanging and can determined and removed from EMmeasurements by applying a corresponding correction value. Static EMdistortion can be caused by a number of factors including objects withinthe operating room that are generally unmoving and thus do not changeduring a procedures as well as larger scale factors such as the magneticfield of the earth. Dynamic EM distortion is caused by factors thatchange over the course of a procedure. These can include any of themoving robotic components of the system, as well as other components (oreven people) who may move through the operating room. Distortiondetection can be used to both detect and correct for static distortionsas well as to find optimal positions for moving component that reducedynamic distortions.

v. Multi-Modal Sensor Fusion

The robotically controlled EM field generator 302 can also be useful inallowing for multi-modal sensor fusion. As used herein, multi-modalsensor fusion can refer to simultaneous or concurrent use of differentsensor types during a single procedure in a synergized way. As anexample, with the EM field generator 302 that is rigidly mounted on therobotic arm 212, EM sensing technology can be registered to otherimaging and sensing modalities mounted on the robotic arm 212 or in therobotic system. Registration of the EM sensing modality with otherimaging and sensing modalities attached to or part of the robotic arm,cart, or base may facilitate, for example, providing a simplifieddisplay (wherein data from a plurality of sensing modalities can bedisplayed together in a unified way), augmented reality, or the like. Inthis section, examples will be described wherein EM sensing technologycan be used with ultrasound technology to provide for multi-modalfunctionality. This may allow, for example, accurate identification of aneedle tip (where the needle tip includes an EM sensor) in ultrasoundimages. In general, determination of the needle tip in an ultrasoundimage may not be straightforward, as artifacts may distort the image.Registering the imaging plane of the ultrasound and the EM field tosingle space, such as the robotic coordinate frame, can facilitatedetermination of the needle tip in the ultrasound image. To achievethis, an ultrasound probe can be rigidly mounted on a robotic arm, aswill be described in more detail below. If the ultrasound imaging planeis kinematically known with respect to the coordinate frame of the EMfield generator 302, the EM sensor inside the needle can be tracked inreal-time and overlaid on the ultrasound image(s).

Attaching an ultrasound probe to a robotic arm can facilitateregistration or calibration of the ultrasound probe's imaging plane.Calibration of an ultrasound probe's imaging plane can generally requirecapturing three different ultrasound images from three different, knownpositions. The three ultrasound images, along with the correspondingknown positions, can be used to calibrate the imaging plane.Conventionally, additional equipment has been required to calibrateultrasound probes used in surgical robotic systems. For example,external position tracking systems would have to be set up to determinethe positions at which the three ultrasound images are captured. As aspecific example, optical sensors (such as infrared LED) can be attachedto an ultrasound probe. A position tracking system, such as Optitrack,can detect the positions of the optical sensors that are attached to theultrasound probe and used to determine the position thereof. In asurgical robotic setting, this can be disadvantageous as additionalequipment is required.

By attaching an ultrasound probe to a robotic arm, the position of theultrasound probe can readily be determined with respect to a roboticcoordinate frame associated with the arm based on the kinematics of thearm in a manner similar to that described above with respect to the EMfield generator attached to a robotic arm. A base ultrasound image canbe captured at this base position. The robotic arm can then move theultrasound probe to a first position at which a first ultrasound imageis captured and the first position can be determined based on thekinematics of the arm. The robotic arm can then move the ultrasoundprobe to a second position at which a second ultrasound image iscaptured and the second position can be determined based on thekinematics of the arm at the second position. The three images (base,first, and second) and positions (base, first, and second) can then beused to calibrate the imaging plane of the ultrasound probe withoutrequiring any external or additional equipment, such as conventionalposition tracking equipment.

FIG. 49A illustrates a robotic medical system with an ultrasonic probe1050 attached to a robotic arm 212 during a procedure for calibrating animaging place of the ultrasonic probe. In the illustrated embodiment,the robotic arm 212 is coupled to the ultrasound probe 1050. Forexample, the ultrasound probe 1050 can be attached to a distal end ofthe robotic arm 212 or an instrument device manipulator (IDM) positionedon the arm. The robotic arm 212 is configured to move to adjust aposition the ultrasound probe 1050. The robotic arm 212 is alsoassociated with a robotic coordinate frame as described previously. Inthe illustrated embodiment of FIG. 49A, three positions Bi, Bj, and Bkare illustrated. The robotic arm 212 moves to move the ultrasound probe212 between these three positions. At each position, the kinematics ofthe arm can be used to determine the position of the ultrasound probe inthe robotic coordinate frame.

At the first position Bi the ultrasound probe 1050 can capture a firstimage. The ultrasound probe 1050 is then moved to the second position Bjusing the robotic arm 212. At the second position Bj a second ultrasoundimage is captured with the ultrasound probe 1050. The ultrasound probe1050 is then moved to the third position Bj using the robotic arm 212.At the third position Bj a third ultrasound image is captured with theultrasound probe 1050. The system can calibrate an imagining plane ofthe ultrasound probe 1050 relative to the robotic coordinate frame basedon the first ultrasound image and first kinematics of the first roboticarm with the ultrasound probe at the first ultrasound probe position,the second ultrasound image and second kinematics of the first roboticarm with the ultrasound probe at the second ultrasound probe position,and the third ultrasound image and third kinematics of the first roboticarm with the ultrasound probe at the second ultrasound probe position.

With the imaging plane calibrated and registered to the roboticcoordinate frame, the robotic system can take advantageous of aplurality of sensing modalities concurrently. FIG. 49B illustrates anexample multi-modal robotic system. As shown in FIG. 49B, a roboticsystem can include a first robotic arm 212A, a second robotic arm 212B,and a third robotic arm 212C. The robotic arms can be attached to acart, a patient platform, or other common structure such that each isassociated with a common robotic coordinate frame. In the illustratedembodiment, the ultrasound probe 1050, for which the imaging plane hasbeen calibrated, is attached to the first robotic arm 212A. An EM fieldgenerator 302 is attached to the second robotic arm 212B. The EM fieldgenerator 302 is configured to generate an EM field within whichpositions of EM sensors, such as the EM sensor 204, can be determined.Using the EM to robotic coordinate frame registration proceduresdescribed above, the position of the EM sensor 204 can be determinedwithin the robotic coordinate frame. Since the imaging plane of theultrasound probe has also been registered to the robotic coordinateframe, the imagine plane the position of the EM sensor 204 are bothdetermined within a common space (the robotic coordinate frame) andthus, can both be displayed on a common display as shown in FIG. 49B.

In FIG. 49B, the EM sensor 204 can be positioned on a distal end of ascope that is inserted into a treatment region of the patient. The scopecan be a robotically controlled medical instrument attached to andcontrolled by a different robotic arm. Alternatively, the scope can be amanually controlled scope. FIG. 49B also illustrates that a medicalinstrument, such a needle can be attached to the third robotic arm 212C.The position of the needle, which can be rigidly attached to the thirdrobotic arm 212C can also be determined with respect to the roboticcoordinate frame. Accordingly, the position of the needle can also bedisplayed along with the imagine plane and EM sensor data. In this way,various sensor technologies can be fused in a single robotic system andused concurrently during a single procedure. In this example, rendezvousof the needle with the scope is facilitated by robotic data, ultrasounddate, and EM data, which can improve the accuracy of rendezvous andprovide a comprehensive and improved operating experience.

FIG. 50 is a flowchart illustrating an example method 1000 forcalibrating an imaging plane of an ultrasonic probe for use with arobotic medical system. The method 1000 begins at block 1001, at whichan ultrasound probe is coupled to a first robotic arm. The first roboticarm is configured to move to adjust a position the ultrasound probe, andthe first robotic arm associated with a robotic coordinate frame. Atblock 1002, the ultrasound probe is moved to a first ultrasound probeposition using the first robotic arm. The first ultrasound probeposition can be a position at which a first ultrasound image to be usedto calibrate the imaging plane of the ultrasound probe will be captured.The first position can be determined based on the kinematics of thefirst robotic arm. At block 1003, the first ultrasound image with theultrasound probe positioned at the first ultrasound probe position. Atblock 1004, the ultrasound probe is moved to a second ultrasound probeposition using the first robotic arm. The second ultrasound probeposition can be a position at which a second image can be captured,which occurs at block 1005. The second image can also be used tocalibrate the imaging plane of the ultrasound probe. The second positioncan be determined based on the kinematics of the arm. As illustrated byarrow 1006, blocks 1004 and 1005 can be repeated to capture a thirdimage at a third position, for example, as shown in FIG. 49A. In someembodiments, these steps can be repeated additional times, capturingadditional images at additional positions.

At block 1007, the imaging plane imagining plane of the ultrasound proberelative to the robotic coordinate frame can be calibrated based on thefirst ultrasound image and first kinematics of the first robotic armwith the ultrasound probe at the first ultrasound probe position, thesecond ultrasound image and second kinematics of the first robotic armwith the ultrasound probe at the second ultrasound probe position, andthe third ultrasound image and third kinematics of the first robotic armwith the ultrasound probe at the third ultrasound probe position. Whencalibrated, the imaging plane can be determined with respect to therobotic coordinate frame and used with other sensing modalities (such asEM) which can also be determined with respect to the robotic coordinateframe.

In some embodiments, the method 1000 may also include coupling an EMfield generator to a second robotic arm, wherein the EM field generatoris configured to generate an EM field and the EM field generatorassociated with an EM coordinate frame, and wherein the second roboticarm is configured to move to adjust a position of the EM field generatorand the second robotic arm associated with the robotic coordinate frame.The method 1000 may also include determining a registration between theEM field coordinate frame and the robotic coordinate frame based onkinematics of the second robotic arm. This can be accomplished asdescribed previously. The first and second robotic arms can be attachedto a cart or a patient platform defining a relationship between thefirst and second robotic arms within the robotic coordinate frame. Thiscan allow the EM field generator and the ultrasound probe to both bebrought within the robotic coordinate frame. The method 1000 may alsoinclude positioning an EM sensor within the EM field, and determining anEM sensor position within the robotic coordinate frame based on theregistration. Additionally, the method 1000 may also include displayingthe imaging plane of the ultrasound probe, and overlaying an indicationof the determined EM sensor position on the displayed imaging plane ofthe ultrasound probe as shown, for example, in FIG. 49B.

Although multi-modal functionality has been described with respect to EMand ultrasound, other modalities can also be used. For example, in someembodiments, a depth sensor can be attached to a robotic arm. The depthsensor can be used to register the position of a patient, a bed orplatform, a fluoroscopic C-arm used during the procedure, and/or otheritems in the surgical space to the global robot reference frame. In someembodiments, these positions can be represented as a heat map or pointcloud, as shown in FIGS. 51A and 51B respectively. A depth sensorcamera, for example as shown in FIG. 51C, can be configured to measurethe distance to objects within a field of view thereof. Calibration forthe depth sensor (for example, to register the output of the depthsensor to a robotic coordinate frame or the global coordinate frame) canproceed in the same manner as calibration of the ultrasound probedescribed above. In general, the depth sensor is calibrated by themanufacturer with respect to pixel and depth information. For use with arobotic system, for example, as described herein, it may be needed tocalibrate the pose of the depth sensor with respect to robotic frame,which as mentioned above, can be accomplished in a similar manner as forthe previously described ultrasound probe.

In some embodiments, a robotic medical system can utilize a plurality ofdifferent sensor functionalities in combination with each other. Forexample, a system can include an ultrasound probe, a camera or depthsensor, and an EM field generator, the output of each of which can becalibrated and registered to the robotic or global coordinate frame. Inthese embodiments, the ultrasound probe can provide imaging datarelative to inside the patient's body. Similarly, the camera or depthsensor can provide imaging or other information about the externalanatomy, for example, the position of the patient as well as externaldevices located in the operating room, such as C-arm. This can beadvantageous as knowing where the patient is with respect to the roboticcoordinate frame can allow the system to identify a needle insertionsite, for example. Additionally, knowing the position of other items inthe operation room with respect to the robotic coordinate frame, such asthe C-arm, can allow the system to know when objects are close to thefield generator, which may cause distortion.

3. Implementing Systems and Terminology.

Implementations disclosed herein provide systems, methods and apparatusfor robotically controllable field generators.

It should be noted that the terms “couple,” “coupling,” “coupled” orother variations of the word couple as used herein may indicate eitheran indirect connection or a direct connection. For example, if a firstcomponent is “coupled” to a second component, the first component may beeither indirectly connected to the second component via anothercomponent or directly connected to the second component.

The phrases referencing specific computer-implemented processes andfunctions described herein may be stored as one or more instructions ona processor-readable or computer-readable medium. The term“computer-readable medium” refers to any available medium that can beaccessed by a computer or processor. By way of example, and notlimitation, such a medium may comprise random access memory (RAM),read-only memory (ROM), electrically erasable programmable read-onlymemory (EEPROM), flash memory, compact disc read-only memory (CD-ROM) orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium that can be used to store desiredprogram code in the form of instructions or data structures and that canbe accessed by a computer. It should be noted that a computer-readablemedium may be tangible and non-transitory. As used herein, the term“code” may refer to software, instructions, code or data that is/areexecutable by a computing device or processor.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isrequired for proper operation of the method that is being described, theorder and/or use of specific steps and/or actions may be modifiedwithout departing from the scope of the claims.

As used herein, the term “plurality” denotes two or more. For example, aplurality of components indicates two or more components. The term“determining” encompasses a wide variety of actions and, therefore,“determining” can include calculating, computing, processing, deriving,investigating, looking up (e.g., looking up in a table, a database oranother data structure), ascertaining and the like. Also, “determining”can include receiving (e.g., receiving information), accessing (e.g.,accessing data in a memory) and the like. Also, “determining” caninclude resolving, selecting, choosing, establishing and the like.

The phrase “based on” does not mean “based only on,” unless expresslyspecified otherwise. In other words, the phrase “based on” describesboth “based only on” and “based at least on.”

The previous descriptions of the disclosed implementations are providedto enable any person skilled in the art to make or use the presentinventions disclosed herein. Various modifications to theseimplementations will be readily apparent to those skilled in the art,and the generic principles defined herein may be applied to otherimplementations without departing from the scope of the invention. Forexample, it will be appreciated that one of ordinary skill in the artwill be able to employ a number corresponding alternative and equivalentstructural details, such as equivalent ways of fastening, mounting,coupling, or engaging tool components, equivalent mechanisms forproducing particular actuation motions, and equivalent mechanisms fordelivering electrical energy. Thus, the disclosure herein is notintended to be limited to the implementations explicitly described butis to be accorded the widest scope consistent with the principles andnovel features disclosed herein.

1. A robotic medical system comprising: a first robotic arm configured to couple to an electromagnetic (EM) field generator and configured to move the EM field generator; and one or more processors configured to: determine an EM position of an EM sensor within the EM field in an EM coordinate frame associated with the EM field generator; determine a position of the EM field generator in a robotic coordinate frame associated with the first robotic arm; determine a registration between the EM coordinate frame and the robotic coordinate frame based on the position of the EM field generator; and based on the registration, determine a position of the EM sensor in the robotic coordinate frame.
 2. The system of claim 1, wherein the one or more processors are configured to determine the position of the EM field generator based on kinematics of the first robotic arm.
 3. The system of claim 1, wherein the EM field generator comprises a compact field generator (cFG).
 4. The system of claim 1, wherein the first robotic arm is configured to removably couple to the EM field generator.
 5. The system of claim 1, further comprising: a second robotic arm configured to control movement of a medical instrument, wherein the robotic coordinate frame is further associated with the second robotic arm.
 6. The system of claim 5, wherein the first robotic arm and the second robotic arm are coupled to a moveable cart.
 7. The system of claim 5, wherein the first robotic arm and the second robotic arm are coupled to a patient platform configured to support a patient during a medical procedure.
 8. The system of claim 5, wherein the EM sensor is positioned on the medical instrument.
 9. The system of claim 1, wherein the one or more processors are further configured to cause the first robotic arm to move the EM field generator to allow access to a patient during a medical procedure.
 10. The system of claim 1, wherein the one or more processors are further configured to: cause the first robotic arm to move the EM field generator relative to the EM sensor to a new position; and determine a new EM position of the EM sensor within the EM field in the EM coordinate frame associated with the EM field generator, wherein the new EM position has improved accuracy compared to the EM position.
 11. The system of claim 1, wherein the one or more processors are further configured to cause the first robotic arm to move the EM field generator to center the EM sensor within a working volume of the EM field generator based on the position of the EM field generator and the position of the first robotic arm.
 12. A method for performing a robotic medical procedure, the method comprising: generating an electromagnetic (EM) field with an EM field generator coupled to a first robotic arm, the EM field associated with an EM coordinate frame; determining a position of the EM field generator within a robotic coordinate frame based on kinematics of the first robotic arm; determining a registration between the EM coordinate frame and the robotic coordinate frame associated with the first robotic arm based on the determined position of the EM field generator; determining a position of an EM sensor within the EM coordinate frame; and based on the registration, determining the position of the EM sensor in the robotic coordinate frame.
 13. The method of claim 12, wherein the determination of the position of the EM field generator within the robotic coordinate frame is based on kinematics of the first robotic arm.
 14. The method of claim 12, further comprising removably coupling the EM field generator to the first robotic arm.
 15. The method of claim 12, further comprising causing the first robotic arm to move the EM field generator.
 16. The method of claim 15, wherein causing the first robotic arm to move the EM field generator comprises causing the first robotic arm to move the EM field generator to a position that allows access to a patient during a medical procedure.
 17. The method of claim 15, wherein causing the first robotic arm to move the EM field generator comprises moving the EM field generator relative to a new position associated with improved accuracy in tracking the EM sensor within the EM field coordinate frame.
 18. The method of claim 15, wherein causing the first robotic arm to move the EM field generator comprises centering the EM sensor within a working volume of the EM field generator.
 19. The method of claim 12, further comprising controlling a medical instrument coupled to a second robotic arm, wherein the robotic coordinate frame is associated with the second robotic arm.
 20. (canceled)
 21. The method of claim 20, further comprising positioning the moveable cart relative to a patient platform configured to support a patient during the robotic medical procedure such that there is a known spatial relationship between robotic coordinate frame and the patient platform. 22-30. (canceled) 